AI Crawler & Answer-Engine Access Policy

1. Public directory pages are open to AI/answer-engine crawlers

The same public registration and marketplace information that's indexed by conventional search engines is also explicitly allowed for AI and answer-engine crawlers, via robots.txt: OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Claude-Web and Claude-SearchBot, Perplexity's PerplexityBot, Google-Extended, Apple's Applebot-Extended, and Common Crawl's CCBot. This list is the same one published in robots.txt itself — if the two ever disagree, robots.txt is the authoritative, machine-readable source.

2. What stays out of reach regardless of crawler

Authenticated areas (admin, staff, employer and agency dashboards, notifications) and private document routes (requirement/employer document downloads, the internal API) are disallowed for every crawler, AI or otherwise — this mirrors what's already kept out of conventional search indexing, not a separate policy for AI specifically.

3. Why this is allowed rather than blocked

The directory content itself is already public: active MEA/eMigrate registration records that anyone can look up directly on the official eMigrate portal, plus marketplace-added profile content an agency has chosen to publish. Blocking AI crawlers wouldn't make that information private — it would only make this marketplace harder to find through the tools employers and agencies increasingly use to research before they search a traditional engine.

4. What allowing a crawler does not mean

Permitting a crawler to fetch a page controls access, not interpretation — it does not mean this marketplace can guarantee how any third-party AI system chooses to summarise, quote or represent that content, and it is not an endorsement of, or partnership with, any AI provider. As stated elsewhere on this site, AI is used on our own side only for appropriate matching and discovery tasks, never to fabricate records — that principle applies to what we build, not to what an external crawler does with a public page once it's fetched it.

5. Reviewing this policy

The allowed-crawler list is reviewed as new, legitimate answer-engine crawlers become established, and tightened if a crawler is found to disregard Disallow rules or misrepresent registration data. Report a concern through the Contact Us page.