Research

ContextualizationIn biology and medicine, viruses are considered to be agents that provide a stress test for the host organism. They can show how robust the immune system is, and where its weaknesses are. The new coronavirus is no exception. Not only has the virus attacked its human hosts, it has also shown that many European governments could not resist the temptation to use the pandemic to increase the powers of the police. And that their immune system – civil society, experts or activists – has often been too weak to prevent their attempts.

Why such an analogy?

There are no policies, strategies, or guidelines on how to implement new technologies for collecting and processing data in Slovenia. The systems for automatic decision-making (ADM) have, therefore, been implemented in many branches of government, such as social services and tax authorities, without much (if any) public debate. Furthermore, whenever the police have attempted to “hack the legislation” – to use any kind of crisis to increase its powers – it has faced only weak opposition from individual privacy experts and activists.

The global pandemic of “the new coronavirus” COVID-19 is the most recent example. During the first wave of the pandemic, the government introduced the Act on intervention measures to mitigate the consequences of COVID-19. But this huge legislative package also included some articles that have significantly increased the powers of the police, which was allowed to collect and process some “special kinds” of personal information, e. g., medical, location, and biometric data (photographic and video images for face recognition).

Furthermore, the government adopted another package of anti-COVID-19 measures in July, which provided a legal basis for introducing the mobile application for contact tracing before the application was introduced and tested According to the new law, the app is obligatory for citizens who are tested positive for the coronavirus or who are in quarantine.

The recent pandemic has shown that the power of the state has increased significantly during the crisis and that the “immune system” was too weak to prevent the new increase of police powers. The European Union should, thus, prepare a common framework that is also legally binding for Member States. Slovenian NGOs are too small and do not have enough expert knowledge to be the principle protectors against state and private misuse of ADM. And Slovenia is probably not the only such example.

In 2010, the Slovenian government introduced the e-Sociala (e-social services) program to optimize social transfers, such as social and unemployment benefits, child benefits, subsidies, etc. that make up the welfare system. The system consists of many connected databases (Ministry of Public Administration, 2009) and was designed to optimize the work process for social workers, evaluate their work, and prevent errors when processing applications for social welfare. We learned from the Ministry of Labor, Family, Social Affairs and Equal Opportunities that the system has been upgraded since its introduction and that it now includes AI, ADM, and machine learning capabilities.

The ministry spokesperson provided several examples of ADM. For example, citizens can use ADM to learn their social rights and see a prediction of benefit payments so that they can make an informed evaluation and calculation before applying for social benefits. Furthermore, the application processes for several social rights/benefits is fully automated, e.g., to research previous decisions related to social benefits and to see new (informative) ones for particular applicants. E-sociala can also detect people from the same family and access their financial information, thereby simplifying the process for social workers.

The goal is to further automate the process so that the system can create a full “social rights profile” for every applicant. The system will then be able to see what benefits a particular citizen is eligible for, for how long, and for what amount of money. For legal reasons, social workers still need to personally evaluate, print, sign and send every application. At the moment, the process still requires some manual (human) intervention which is recorded and analyzed in order to improve the system and prevent errors.

The ministry admits that some human input will always be necessary because of changing and ever more complex life scenarios, and the quality of the input data. However, they hope to automate their system to such a level that social workers will only need to intervene in very specific situations.

In May 2019, the former Slovenian airline company, Adria Airways, announced (STA, 2019) that they had successfully tested a pilot project on biometric boarding together with Polish airline operator LOT.

The two airline companies were a testing ground for the US IT company, Amadeus, which is developing and implementing a large centralized database of biometric data for the airline industry. The biometric boarding system helped 175 passengers to board in “a record short time”. Boarding took only two seconds per passenger instead of five to seven seconds, according to the press release.

The passengers were asked to install the Amadeus smartphone app, take a selfie, photograph their passport (including their portrait photograph), and upload the data to a remote server. The passenger photographs were then captured at the time of boarding, and the system automatically verified each individual. Following a positive match, a confirmation message was sent to the boarding system, and the passenger was allowed to board. All the biometric data used during the test was deleted within 48 hours of the test, the Adria representatives said.

During the research for this report, we learned that Amadeus has signed a partnership agreement with Ljubljana airport, which is operated by the German company, Fraport. Amadeus helped Fraport to upgrade its information systems, and biometric boarding was also a part of this update. The system should have been fully functional by the end of 2019. However, there was no additional information available at the time of writing.

An increasing number of Slovenian cities, and local municipalities, have joined the “smart city” initiative. These cities include Maribor, Kranj, Ljubljana, and Novo Mesto, among others.

However, there are many problems with these “smart city” projects. The state provided no strategy or guidelines on how to implement “smart” solutions (like ADM) and it has no oversight on what is happening at the local level. Furthermore, some local municipalities seem to have little or no knowledge of how their systems are designed and operated, who hosts and analyzes the data, and whether or not there are any ADM systems, etc.

Furthermore, “smart city” initiatives are still relatively unsophisticated. For example, the most ambitious “smart city” plans were announced in the city of Novo Mesto in 2018 (Rajšek, 2018). The city authorities announced that – in collaboration with the German IT company, SAP, and the Slovenian national telecommunications operator, Telekom Slovenije – Novo Mesto would become the “first reference smart city” in Slovenia. Among the “smart” services were the real-time measurement of air quality, availability of public parking spaces, traffic analysis, intelligent street lighting systems, and the measurement of public opinion (i.e. the public’s satisfaction with public services).  The city is also testing IoT (Internet of Things) solutions to gather and analyze data more effectively.

However, according to a spokesperson for Novo Mesto, this “first reference smart city” is not using any ADM or AI systems as the city officials are still trying to find practical use of the data.

In 2018, Slovenian NGO Državljan D started a campaign (Državljan D, 2018) to document online ads from big Slovenian advertisers – including state and state-owned companies – that appeared on various media websites, and regularly featured problematic, hateful, and false articles on migrants, ethnic, and religious minorities (Muslims, LGBTQ members, etc.). Državljan D argued that advertisers were financing hate speech and called on companies to stop advertising on such websites.

At the beginning of the campaign, the advertisers tried to blame “advertising algorithms” for the problematic placement of their ads and they denied any responsibility. However, what looked like an anecdotal example of ADM turned bad, soon became a political (and politicized) story. In November 2018, Slovenian Prime Minister Marjan Šarec joined the debate when he published a public statement calling for advertisers to reconsider their marketing strategies when their clients promote hate speech (e.g., racism and homophobia).

However, state and state-owned companies continue to advertise on websites that promote hate speech. Among such companies are the national incumbent telecommunications operator Telekom Slovenije, Slovenian Railways, and some other companies where the state has some influence over the advertising budgets. Meanwhile, financing hate speech through (state) advertising has become a regional and international problem. The Slovenian investigative medium Pod črto published a series of articles about Slovenian media (Kučić, 2019) and learned that political actors have built a parallel media system that also includes a network consisting of 17 regional Internet portals.

Since 2017, the portals have published more than 20,000 articles. Most of the content is neutral, but the anonymous authors also publish a lot of articles about migrants and other political (or politicized) topics. This content is then promoted on social media platforms to target individuals on their profiles.

The Hungarian media businessmen close to the Hungarian political party Fidesz and Prime Minister, Viktor Orban, who is expanding his party media empire to neighboring states to support his political allies according to many cross-border investigations by journalists ((see, for example, Cvetkovska, Zöldi, and Delić, 2020).

Similar attempts have also been documented in other countries in the region, including Macedonia, Hungary, and Poland, among others. State advertising and the money from state-owned companies have been used – sometimes illegally – to promote political agendas of the ruling political parties and influence public opinion. Playing with algorithms on social media platforms has become an essential part of the game. And this “game” also includes targeted attacks on the political opposition, journalists, NGO’s, and private individuals, according to media reports.

Since we included the story in our first report, the focus of this example has shifted from ADM to computational propaganda. But it should be seen as a warning, because some recent examples have shown (e.g., The Cambridge Analytica case) that the use of big data, machine learning, and targeted advertising (which is often automated) has a strong potential to distort public opinion. Especially when the government or a political party can finance such an operation with state and public money.

The Ministry of Finance’s financial administration has been using machine learning to detect tax evasion schemes, and tax fraud, and to find errors in tax reports. During the research for this report, we also learned that the ministry ranks “risky” citizens who it thinks are more likely to become tax evaders.

This year, we learned about some additional information about this system. In 2017, the financial administration started using a fraud management software system, produced by SAP. The administration also formed working groups to find the most likely scenarios for tax evasion and tax fraud. The groups were asked to define the indicators of risk that may help detect illegal activities. They listed 158 risk factors and used them to select around 17,500 individuals and companies for tax inspection. The inspectors found irregularities in more than 75% of all the selected cases.

The system proved to be a useful tool for tax inspectors, according to the spokesperson. The tax authority would not give us specific examples on how they are using their prediction models because they would not want help future tax-evaders to “game their system”.

The spokesperson explained that every new tax evasion indicator (alert) is instantly shared with all users of the system (tax inspectors). The system can also reduce the risk of tax inspectors deliberately avoiding more demanding or “untouchable” cases. However, a spokesperson for the financial administration explained that fraud detection cannot be fully automated. The quality of data and the rapid evolution of tax-evasion systems being the biggest obstacles for a tax ADM.

Slovenia was one of the first countries in the world to develop and test an intelligent system to detect early learning problems in primary schools (METIS). The system was designed to monitor pupils’ grades and absences, to search for specific learning patterns, and help teachers find “problematic” pupils. Unfortunately, METIS was also an example of how not to implement an ADM process into the educational system.

Firstly, the budget was too small. The Ministry of Education, Science and Sport allocated only 70,000 euros for the project. Thus, the researchers had to take many shortcuts. They apparently used “interpolated” data, together with the “real data”, to build a detection model. The educational experts were also critical of the methodology and they claimed that grades and absences by themselves were not sufficient indicators to reliably detect learning problems. “You do not need AI to see that a pupil has got worse grades in the second semester. You can do that with Excel,” said a speaker at the Grounded festival.

The ministry argued that they never endorsed METIS as an official tool for detecting learning and behavioral problems. However, the public debate also addressed another related issue: that a lot of school data is collected and owned by a private (commercial) contractor. And that the contractor is probably the only institution in Slovenia capable of building an efficient ADM system for educational purposes. Or to provide personalized profiles of pupils for future employers and universities that could predict their performance as workers and students.

In the first edition of this report, we learned that Slovenia had not adopted any strategic document at the national level regarding the use of AI, algorithms, or ADM systems.

That has not changed. However, according to a spokesperson, the Ministry of Public Administration has initiated preliminary activities that will eventually lead to a national AI strategy. The ministry formed a cross-sector working group that was expected to present a first draft of the strategy by the end of February 2020 (the draft was still not published at the time of publication). The ministry also invited representatives from industry groups, research institutions, and NGOs to join the working group. In addition, the ministry states that it wants to establish an efficient model of support for research, development, and implementation of AI solutions by 2025.

In October, the Slovenian mission to the UN, the Council of Europe, and UNESCO hosted an event to discuss the challenges of AI at the UN headquarters in New York. The debate focused on the legal and ethical issues of existing and future uses of AI. Slovenia’s Ambassador to the UN, Darja Bavdaž Kuret, said in conclusion that Slovenia would remain at the forefront of discussions on AI, according to a report by the Slovenian Press Agency STA (STA, 2019).

Despite these bold claims at the UN, AI and ADM have hardly been “at the forefront” of national discussions in Slovenia. Only a few events have attracted public and media attention.

In November 2019, UNESCO’s general conference decided that the first UNESCO-sponsored international center for AI will be located in Ljubljana. The Department of Intelligent Systems, at the Jožef Stefan Institute (JSI), will transform into a center that focuses on governance and policies surrounding AI. Furthermore, the International Research Center for Artificial Intelligence (IRCAI) was also established in May 2020, said the spokesperson for the JSI.

The new facility will “aim to provide an open and transparent environment for AI research and debates on AI, providing expert support to stakeholders around the globe in drafting guidelines and action plans for AI,” according to the Slovenian Press Agency (STA, 2019). The center will advise governments, organizations, legal personnel, and the public on systemic and strategic solutions related to the introduction of AI in various fields. It will also run consultations with the wider public about the impact of AI.

The new center will retain its existing research staff and funding structure. It currently has an annual budget of around €500,000 from a combination of national and EU funds (Banovic, 2019).

Automation and Power was the central theme of the Grounded festival in November 2019 (Grounded, 2019). The festival addresses current social issues by combining theoretical debates, social activism and progressive electronic club culture.

Speakers at the festival presented some examples of ADM that were mentioned in the first edition of this report. Among these examples were a system for detecting children with potential learning and educational problems and another for predicting potential tax evaders. In many cases, it was the first time that discussions, Q&A sessions, and presentations on these topics had happened in public. It was also the first time that state representatives – from the Ministry of Education, the financial administration, and other departments – had to answer some hard questions about the current and future uses of AI and ADM.

Such opportunities are important. The speakers and organizers agreed that Slovenian civil society rarely has any insight into the operation of such systems, let alone any influence on the desirability of their introduction. They believe that the political, social, economic, health, justice, and education systems can no longer be discussed without a simultaneous debate on the automation that underpins their development. Instead, automation (as a process) is shying away from democratic scrutiny due to the complexity and specialization of knowledge, high financial input, and related interests.

In addition, Slovenia is in the process of implementing a 5G infrastructure, under pressure from the EU,  and without any public participation, following the mandate of ever-faster and more data-rich connectivity, warned the Grounded festival organizers.

What can AI learn from dogs? This is a question of practice-oriented, artistic research aimed at developing an algorithm for artificial neural networks co-programmed by dogs.

Brute Force is one of the first art projects in Slovenia to look at aspects of AI that challenge our social and political reality (Ars Electronica, 2019).

The artist, Maja Smrekar, who started Brute Force, explained (to us) that the algorithms for automating the societal processes often follow behavioral doctrine once used for training dogs. Technology companies rely heavily on behavioral experiments by the late psychologist B. F. Skinner. They use social constructivism and social conditioning to influence human behavior – to either buy a product, vote for a particular political candidate or change our mood. Ironically, dog breeders and trainers are abandoning such behavioral approaches and conditioning that are now being applied to humans, because they destroy the dog’s “personality”.

Smrekar uses the analogy with dog training to unveil this behaviorist regime and its possible social impact. Can algorithms predict a dog’s behavior, and how can we use such information? To learn from it, and to train them better and influence their behavior? Can algorithmic prediction be used to control and enslave humans as feared by some contemporary thinkers like Yuval Noah Harari? Such futures are possible, Maja Smrekar believes. But resistance to such conditions also requires technological understanding, which might lead to turning the system’s logic against itself and to the emancipation of humans.

The absence of a national strategy and public debate on AI and ADM are the two defining factors that influence the use and implementation of IT solutions for automating Slovenian society.

There is a global trend towards automation, and Slovenia is no exception. Slovenian companies, ministries, public institutions, and local authorities are buying and testing all kinds of solutions that use AI and ADM, but nobody knows the full extent of such testing and implementation.

In addition, there are many inconsistencies. On the one hand, the politicians and state officials emphasize the role of AI and ADM for the future of society and the economy, but they are unwilling to finance research and development in this field. Furthermore, they are vocal about humane uses of AI, social equality, and respect for human rights, but they ignore all the warnings against the increasing economic inequality and the potentially unconstitutional powers of the police to collect personal information and monitor Slovenian citizens secretly or even illegally.

Consequently, the state plays a retroactive instead of a proactive role. The government, and regulators, react when something goes wrong or catches public attention, but they fail to start solving problems that could most likely be both predicted and prevented. In addition, too much pressure is put on NGOs and privacy activists to protect the citizens from the police surveillance infrastructure because they do not have enough staff, specialized knowledge, and funding for such a task. Nor can they rely on European institutions to provide a legal framework and political pressure to the Member States that have shown repressive tendencies during the pandemic.