Sovereign AI in Poland: Language Adaptation, Local Control & Cost Advantages with Marek Kozlowski

Kozlowski explains the core thesis of localized LMs: creating models 10x smaller than frontier models but with equivalent quality in specific languages and domains. He discusses how 90% of training data is English/Chinese, leading to cultural and linguistic gaps in other languages, and why models trained primarily on English produce non-native sounding text in Polish despite being technically correct.

• •Goal: Build models 10x smaller than frontier models with equivalent quality in Polish language and culture
• •90%+ of training data in major models is English and Chinese, leaving only ~1% for languages like Polish
• •Frontier models can be communicative in Polish but produce non-native phrases (e.g., translating English email formulas word-for-word)
• •Current benchmarks (like MMLU) test understanding and extraction but not fluency in generating longer-form native text
• •Transparency is key: PLLuM publishes ~100-page cookbook, sample datasets, and instructions - not just open weights

Discussion of how EU AI Act and copyright regulations create much harder constraints than US companies face, forcing European teams to rely on curated datasets rather than massive web scrapes. Kozlowski explains the data curation process including deduplication and filtering that can reduce datasets by 50-80%, and why they don't have the trillion tokens needed for training from scratch.

• •EU AI Act and authorship rights regulations can eliminate 80% of potential training data
• •US companies can crawl Polish websites even when disallowed by robots.txt, harder to prosecute internationally
• •Data curation involves deduplication (can reduce by 2x) and filtering out low-quality/corrupted text
• •Poland has several hundred billion tokens after curation, not the trillion+ needed for training from scratch
• •Access to non-web data sources (libraries, bilateral publisher agreements) provides only a small fraction of total data

Kozlowski details PLLuM's competitive advantage: dozens to hundreds of human annotators creating organic instructions and preferences rather than relying on synthetic data. He explains why synthetic instructions from other LLMs can degrade model quality if linguistically poor, and how they've built internal tools rather than using commercial annotation platforms.

• •Competitive advantage is organic (human-created) instruction and preference data, not pre-training data
• •Synthetic instructions from other LLMs can degrade linguistic quality if not carefully reviewed
• •Built internal annotation platform rather than using Scale AI or similar services
• •Human review and correction of LLM-generated instructions maintains quality
• •Published samples of instructions/preferences to demonstrate structure while keeping most proprietary

Technical explanation of how PLLuM uses language adaptation rather than training from scratch. They take LLAMA or Mistral base models and continue pre-training on Polish corpora, creating new base models that retain some English capabilities while gaining Polish fluency. Discusses the forgetting problem in cascade learning and why 1 trillion tokens are needed for stable 8B parameter models.

• •Use LLAMA and Mistral base models, continue pre-training on ~200B Polish tokens (language adaptation)
• •Research shows 8B parameter models need 1 trillion tokens for stable training from scratch
• •Cascade learning causes some forgetting of previous capabilities but doesn't completely prune them
• •Models retain English competency after Polish adaptation, though some degradation occurs
• •Experimenting with mixture of languages to reach 1T tokens for potential training from scratch

Critical finding: Anthropic Claude and OpenAI GPT models are showing declining performance in Polish language and cultural competency across successive releases. As companies optimize for high-value use cases like coding, they're deprioritizing niche language quality, creating integration risk for businesses that depend on these models.

• •Claude and GPT models showing declining Polish language/cultural competency in newer versions
• •Frontier labs prioritizing software development assistance over niche language quality
• •Creates vendor lock-in risk: companies integrate cloud LLMs, then quality degrades in later releases
• •Trade-offs in model development: improving some competencies can degrade others
• •Validates need for locally-controlled models that won't deprioritize specific language markets

Kozlowski explains domain adaptation (continued pre-training on company-specific data) and its requirements. Working with major European bank PKO BP, they found you need 10B+ tokens after deduplication (30-40B before) to make domain adaptation worthwhile - limiting this approach to only the largest companies with sufficient proprietary data.

• •Domain adaptation requires minimum 10B tokens after deduplication/filtering
• •Need 30-40B raw tokens before curation to reach 10B usable tokens
• •Successfully performed domain adaptation for PKO BP (largest bank in Central/Eastern Europe)
• •Fewer than 100 European companies likely have sufficient data volume for domain adaptation
• •Email, contracts, and internal documents add up slower than expected when properly curated

Strategic argument for why AI's future favors small, specialized models over massive general-purpose ones. Businesses typically need 10-20 use cases, not 1000+ tasks. With 1000+ training examples per task, supervised fine-tuning of smaller models achieves same quality as few-shot prompting of large cloud models, with better cost, control, and regulatory compliance.

• •Businesses need 10-20 specific use cases, not general-purpose 1000+ task capability
• •1000+ organic training examples per task enables smaller models to match cloud LLM quality
• •On-premise requirements (government, regulated industries) force GPU/energy cost optimization
• •Agentic AI systems with multiple specialized models favor small, efficient models over large ones
• •Supervised fine-tuning of small models often outperforms few-shot prompting of large models for specific tasks

Unlike US companies developing detailed AI constitutions, Poland hasn't needed explicit value articulation yet. Current focus is on ethical behavior and data usage regulations rather than behavioral constraints. Kozlowski notes they're a few years behind US/China in development, with regulations focused on what data can be used rather than how models should behave.

• •No formal AI constitution defining Polish values for AI systems yet
• •Focus on basic ethical behavior: avoiding hate speech, rudeness, blame
• •Regulations emphasize data usage rights rather than model behavior constraints
• •Release base, instruct, and chat models to let companies choose their own constraints
• •May develop constitution in 1-2 years as models become more capable and widely deployed

Poland's government is prioritizing GPU infrastructure (AI factories) over competing with US salaries for top talent. Kozlowski explains the challenge of competing when US AI engineers earn 'NFL quarterback' salaries, and why public sector projects struggle to define monetizable objectives that would justify such compensation.

• •Primary investment focus: GPU infrastructure and data centers (AI factories)
• •Cannot compete with US salaries where top AI engineers earn like 'NFL quarterbacks'
• •Public sector struggles to define clear ROI metrics that justify premium talent compensation
• •PLLuM project provides some talent funding through university/institute consortium
• •Chinese companies now matching US compensation, creating three-way talent competition

Kozlowski observes that frontier model improvements are becoming more incremental (plateau effect) rather than the shocking leaps of 2022-2024. He argues the industry is entering a verification phase where businesses will evaluate real costs and benefits, likely concluding they don't need massive general-purpose models for their specific use cases.

• •GPT-5 not showing huge improvement over GPT-4; progress becoming more horizontal than vertical
• •Industry entering 'verification stage' to evaluate real costs vs. benefits of AI tools
• •Most business cases don't require reasoning capabilities, can use standard LLMs
• •Businesses need to define specific requirements and benchmarks before choosing models
• •Disconnect between AI development pace and business adoption/evaluation cycles

Discussion of the fundamental tension between distributed cooperation (ideal for spreading wealth and knowledge) and centralization (necessary for fast outcomes). Kozlowski uses Silicon Valley as example: while distributed startups across US would be economically better, concentration enables rapid innovation. Questions whether US-China rivalry allows for cooperation.

• •Fundamental trade-off: federation/distribution is ideal, but centralization delivers faster results
• •Silicon Valley concentration enables speed but creates geographic inequality
• •European projects attempt international collaboration but struggle with coordination overhead
• •Mistral (European startup) now 30-40% owned by Microsoft, losing independence
• •Open question whether US-China AI competition allows for any cooperation or remains purely adversarial