This position paper discusses system-level defenses against indirect prompt injection attacks in AI agents, emphasizing dynamic replanning, context-aware security, and the importance of human interaction in design. It critiques existing benchmarks and promotes integrated security measures to enhance the robustness of AI systems.