“I, Robot” – The European Commission Puts the Breaks on Artificial Inteligence
Text by journalist Tanya Petrova for SEGA, created as part of the project “Digital Democracies vs. Digital Dictatorships” (ACF/970) by the Bulgarian Center for Not-for-Profit Law, funded by the Active Citizens Fund Bulgaria under the EEA Financial Mechanism 2014–2021.
Artificial intelligence raises legitimate concerns that can easily grow into irrational fears.
The European Union is on track to adopt the world’s first legislation aimed at protecting human rights from artificial intelligence.
The world is facing an unprecedented technological leap. AI is set to rapidly reshape the foundations of entire societal systems. It feels as if the boldest futuristic scenarios are coming to life before our eyes. Many boundaries are being tested, and ethics has returned to the center of public debate.
These boundaries can be crossed easily and subtly, and the fear of algorithms making decisions instead of humans is already a reality. At this crucial moment for humanity, Europe is preparing to adopt the first-ever legislation to safeguard people from the risks of AI. The draft regulation, which sets common rules for all EU member states, is in its final stage after more than three years of discussions and preparatory work. The EU aims to hold the ethical “reins” on this rapidly developing sector without hindering it through excessive regulation in the global economic race.
Bulgaria will also have to develop a system for applying ethical standards in the development and implementation of AI systems. What principles underpin these new regulations, and how well do they address the risks posed by algorithms?
Will humanity lose control over decision-making?
We have long relied on machines, especially on computers, and each technological revolution has shifted the tectonic plates of our society. Experts, however, warn that AI carries a unique risk, which could lead to humans losing control over decision-making. “A major difference between AI and traditional information systems is that with AI, it’s not always possible to explain why a model made a specific decision or what input factors led to it,” say the authors of Bulgaria’s AI Development Concept 2030. For this reason, AI algorithms are often compared to “black boxes”, which on top of that, can self-learn by using vast datasets. The risks posed by this opacity, complexity, and dependence on big data are serious. Manipulation of data or decision-making mechanisms is possible without the user ever realizing it.
The list of fundamental rights that can be violated by AI systems is long. It includes the right to dignity, privacy, and personal data protection, the presumption of innocence, the rights of children and persons with disabilities, etc. Freedoms of expression and assembly may also be compromised.
Protecting these core rights is at the heart of the EU’s draft regulation, particularly after it became clear that some member states were preparing their own national legislation, posing a serious risk of legal fragmentation and internal market disruption. Harmonized rules are expected to be more effective and to build public trust.
Can the EU regulate a rapidly expanding key sector?
The draft regulation is based on 1,215 stakeholder inputs and more than three years of consultations. Based on this feedback, experts opted for a risk-based approach, where only high-risk AI systems are subject to binding regulation. Low-risk systems will be subject to non-binding guidelines. This model was chosen over four alternatives to ensure effective rules without burdening the sector with overregulation.
There are four categories of AI systems that the EU plans to ban outright:
- Systems that deliver subliminal messages to influence human behavior.
- Systems that exploit the vulnerabilities of children, the elderly, or persons with disabilities.
- Systems that assess trustworthiness based on social behavior.
- Remote real-time biometric identification systems, except in narrowly defined cases such as searching for missing children or responding to terrorist attacks.
In addition, the European Commission proposes nine more high-risk sectors (listed below), where AI systems must meet strict certification requirements.
All of these systems must include risk management mechanisms, including threat analysis and mitigation measures. They must have the capacity to automatically record events to ensure traceability. Transparency is also a key requirement – users must receive information about the provider, the system’s capabilities and limitations, and in some cases, the training data used.
Human oversight is also mandatory for high-risk AI systems. Operators must be able to override system outputs, stop the system via a “kill switch,” and more. Real-time biometric identification systems may be used only under exceptional circumstances and must be monitored by at least two human operators. The regulation includes a wide range of requirements, from development to market deployment, focused on ensuring transparency and accountability throughout.
Implementation lies in the hands of national governments
Enforcing all these requirements at the national level won’t be easy. It will require administrative capacity and the creation of new institutions. High-risk systems must be assessed and certified by independent evaluation bodies, which will operate under the supervision of national authorities. These authorities must inform the European Commission and other member states via a special platform about any upcoming evaluations of high-risk AI systems. The notification must contain full details about the system and the planned evaluation. The process can begin only if the Commission raises no objections. Evaluations will use harmonized standards or common specifications, and compliant systems will be valid for up to five years and will receive a CE certificate.
All of this will require new national legislation. Bulgaria has already adopted a Concept for the Development of AI until 2030, but work on a national action plan and new legal framework will only begin after the European regulation is passed. Although the European Commission has been working on this since 2018, Bulgaria is lagging behind: “The government has pledged to begin a meaningful dialogue on developing standards to ensure equal access and human rights protection,” said Zahari Yankov, legal expert at the Bulgarian Center for Not-for-Profit Law (BCNL), during the recent roundtable “Artificial Intelligence – Regulation and the Future” at the Council of Ministers.
This measure is part of Bulgaria’s Fourth National Action Plan under the Open Government Partnership. Although adopted in 2022, it remains largely unimplemented. The plan envisions involving human rights organizations in the AI Working Group under the Ministry of e-Government, to ensure transparency and broad public participation in future standard-setting.
“We speak of digital democracy because every new topic that raises ethical and value-based dilemmas requires inclusive conversation,” BCNL notes. “Such public dialogue must include as many affected people as possible—whether or not they fully understand how the technology works.”
BCNL insists that the right to protection must be ensured through explicit legal mechanisms, including:
- Access to AI systems for vulnerable groups;
- The right to file complaints for AI-related rights violations;
- The right to legal representation.
These demands are echoed in a joint statement by 75 major European civil society organizations.
The European Parliament is expected to vote on the regulation in June.
High-Risk AI Systems (as identified by the EU)
- AI systems that form part of safety-critical components subject to regulation.
- Biometric identification and categorization systems in real time or retrospectively.
- Systems determining access to education or evaluating student performance.
- Systems managing or operating critical infrastructure.
- Employment-related AI systems used for recruitment, promotion, performance evaluation, or dismissal.
- Public systems assessing eligibility for services, emergency aid, or credit scoring.
- AI in law enforcement.
- AI used in migration, asylum, and border control.
- AI in the judicial system and democratic processes, including legal fact-finding and law application.
Historian Yuval Harari: “We’re Facing a New Weapon of Mass Destruction”
Concerns about AI have existed since the dawn of the computer era. Until recently, they focused on physical threats machines that kill, enslave, or replace humans. But over the past two years, a new threat has emerged: AI’s manipulation of language. AI has developed powerful capabilities for processing and generating language through text, sound, and imagery. In doing so, it has “hacked the operating system of our culture,” writes Yuval Harari in a powerful op-ed for The Economist. Harari warns that AI is a new weapon of mass destruction, and we must respond as we did to nuclear technology.
“Since 1945, we’ve known nuclear energy can serve or destroy humanity. So we reshaped the global order to keep it in check. Now we must do the same with AI, a weapon that could destroy our mental and social worlds.
We can still design beneficial AI tools, Harari says, but we must act fast. While nuclear weapons can’t create stronger nukes, AI can produce ever-more-powerful versions of itself.
“The first critical step is mandatory safety checks before releasing powerful AI into the public domain. Just like pharmaceutical companies can’t release drugs without testing their side effects, tech companies shouldn’t release AI tools until they’re proven safe. We need a ‘Tech FDA’—and we need it yesterday.”
“Democracies won’t fall behind authoritarian regimes by slowing AI development. Quite the opposite. Unregulated AI could cause social chaos that benefits authoritarianism and undermines democracy. Democracy is a conversation—and conversations rely on language. If AI hacks language, it could destroy our ability to speak meaningfully, and with that, destroy democracy.”
This project is funded by the Active Citizens Fund Bulgaria under the EEA Financial Mechanism. All responsibility for the content lies with the Bulgarian Center for Not-for-Profit Law and the Association of European Journalists – Bulgaria, and under no circumstances can it be assumed to reflect the official position of the EEA Financial Mechanism or the Operator of the Active Citizens Fund Bulgaria.
