The new auto eval tool from Anthropic presents both pros and cons based on a live testing session. It's noted for its ability to identify issues with auto-evaluation that others have missed, but it also forces users into eval creation without adequate data review, lacks context for validation, and bundles too many checks into one evaluator. More flexibility and improved data exploration workflows are recommended before fully adopting the tool.