Arboriste

Katalog arboristů (https://katalogarboristu.cz) — a catalogue of Czech arborists: who they are, which certificates they hold and until when, where they work and how to reach them. A sibling of Stromopedie, in the same forest look.

Blazor Web App (interactive server) on .NET 10 with MudBlazor. The data comes from a crawler that runs inside the app once a week; the same crawler is available as a command-line tool.

src/
  Arboriste.Crawling/   sources, polite HTTP fetcher with a gzip cache, merging, output files
  Arboriste.Crawler/    command-line front end (arboriste-crawler)
  Arboriste.Web/        the catalogue web app; hosts the weekly crawl

Sources

The EAC portal (portal.eac-arboriculture.eu) runs on the same database as the ČCA register, so its Czech entries already arrive through cca. The regional catalogues idatabaze.cz, netkatalog.cz, netfirmy.cz and ekatalog.cz refuse the crawler (HTTP 403) and are not used.

Running locally

dotnet run --project src/Arboriste.Crawler     # crawl into ./data (about 12 minutes, cached for 7 days)
dotnet run --project src/Arboriste.Web         # http://localhost:5171, reads ./data

In Development the web app reads ../../data and does not crawl on its own (appsettings.Development.json). Crawler options: --source cca,firmy, --refresh, --out <dir>, --firmy-tags arborista,rizikove-kaceni, --verbose; merge re-merges the raw files without crawling.

• / — catalogue with search and filters (region, certificate, service, company or person); the filter lives in the query string, so a filtered list can be shared
• /arborista/{id} — detail: certificates with validity, certified people, services, contacts, sources
• /?zobrazeni=mapa — the same catalogue on a map (pins for arborists with an address; clicking a region filters)
• /o-katalogu — where the data comes from, how fresh it is, how to have an entry removed
• /health — { "status", "version", "name", "entries": { "catalog", "map", "admin" } }, unauthenticated
• /admin — one password (Admin:Password), a month-long cookie: overview of the catalogue and the last crawl, Ceníky (price research) and Logy (live log events and the daily log files)

Data folder

App_Data in the container (/opt/arboriste/data on the VPS), ./data locally — git-ignored, it holds personal contacts:

The weekly crawl

CrawlScheduler runs the crawl inside the web app every 7 days at 3 a.m. Prague time (Crawler:Interval, Crawler:Hour), and right away on a fresh install with no catalogue. It costs no extra container or cron job; while it runs the app uses a few tens of MB more.

• One request at a time per site, 1.5 s apart (2.5 s for firmy.cz); the sources run side by side.
• A site that answers 403/429 or with a CAPTCHA page stops that source. The crawler does not disguise itself or work around a block; the source's previous data is kept, and an interrupted run never overwrites an earlier complete result.
• /health turns Degraded when the catalogue is older than three intervals.

Maps

• Base map: Mapy.com tiles through /api/tiles/{z}/{x}/{y}, which adds the key (Mapy:ApiKey) on the server, the same way as boudakaci.cz. Without a key, Development falls back to OpenStreetMap and Production shows no base map (/health → Degraded). Leaflet 1.9.4 is vendored in wwwroot/vendor/leaflet.
• Region and district boundaries: src/Arboriste.Crawling/Data/territory.geojson — 14 regions and 77 districts from ČÚZK INSPIRE Územní správní jednotky (CC BY 4.0), simplified to ~200 m (430 kB, 117 kB gzipped). Rebuild with node tools/build-territory.mjs. The crawler uses it to put a seat into its district and region; the web serves it at /data/territory.geojson.
• Geocoding: with a key, addresses without coordinates (mostly Stromy pod kontrolou) are geocoded once with Mapy.com; answers are kept in geocode.json in the data folder. The CLI reads the key from ARBORISTE_MAPY_KEY.
• Pins are rounded to ~100 m: an address of a sole trader is often a home.

Arborists manage their own entries

• Taking over an entry (/spravovat/{id}, linked from every detail): a one-time link (30 minutes) goes to an e-mail the entry already lists — it comes from the registers and directories, not from the claimant, so reaching that inbox is the proof. The addresses are shown masked. Nobody reachable that way asks the admin instead (their own e-mail is confirmed first); the admin checks them, e.g. by phone, and hands the entry over.
• Signing in (/prihlaseni): by e-mail link only; no passwords are stored. Links are stored as hashes, used once, and at most four go to one address per hour.
• Editing (/muj-zaznam/{id}): name, IČO, contacts, address, districts (okresy) the arborist works in, services, description. Edits wait for the admin (/admin/zaznamy shows only the changed fields); hiding the entry takes effect at once. Certificates are not editable — they come from the registers only.
• New entries (/pridat): after confirming the e-mail; published at once. The form points to existing entries with the same IČO, e-mail or phone and suggests taking one over instead.
• How it joins the catalogue: the profiles live in arboriste.db (SQLite) and are written as the source self (raw/self.json) next to the crawled ones; every publish merges the catalogue again. A profile joins crawled records only through the references it took over (LinkedSources), never through shared contacts, so a new entry cannot take over someone else's by quoting their IČO. What the arborist filled in wins; the entry keeps its URL.
• E-mail: Resend (Email:ResendApiKey, Email:From). Until it is configured every message waits in the admin outbox, where a link can be copied and passed on by hand.

Price research (admin)

/admin/ceniky (or arboriste-crawler prices locally) visits the website of every provider, reads the home page, pages and PDFs called ceník / ceny / sazebník (else the service pages), and pulls out every amount in Kč with its unit, VAT note and a guessed service. Each price keeps the raw block of text it came from, and the whole text read from each page is kept too (/admin/ceniky/text/{key}), so every number can be checked against its source. Results: prices.json and prices/pages/ in the data folder; CSV at /admin/ceniky.csv. Directory profiles (nejremeslnici.cz, …) are skipped; hosting firewalls that answer 401/403 are reported as blocked, not worked around. This is research for the admin only — nothing of it is shown on the public site.

Merging

Records are grouped when they share an IČO, an e-mail, a phone number, a website domain (not free-mail or social domains), or — for people — the same name without academic titles. A group with a company record or several different people becomes a company with its certified people listed inside.

Deployment

Azure Pipelines → Docker image → VPS behind the shared nginx, the same way as Stromopedie. See docs/deployment.md.