ASP MVC routing 404 and images issue

I have a problem with my routing. I am trying to prevent 404 errors to end up in my custom routeconstraint.

This is some of my routes:

        routes.IgnoreRoute("Content/{*pathInfo}");
        routes.IgnoreRoute("{*allfiles}", new { allfiles = @".*\.(css|js|gif|jpg|png)" });

        routes.MapRoute(
            name: "MenuRoute",
            url: "{*permalink}",
            defaults: new { controller = "Menu", action = "Index" },
            constraints: new { permalink = new MenuUrlConstraint() }
        );

My issue is that everytime a 404 appears for static files they go to my MenuRoute.

Is there a way for me to avoid this happening?

Or is the only solution to check for 404 in my routeconstraint, and do nothing if so?