As the previous Automating Society report stressed, the Swedish government is strongly invested in placing the country at the forefront of the development of artificial intelligence (AI) and automated decision-making (ADM).
The previous report addressed the AI initiatives that have been in place since 2017, including government reports on self-driving cars and an AI commission. These initiatives continue to function, but not all of them have been extended.
In 2018, a new public agency for digitalization (DIGG – Agency for Digital Government) was established to coordinate general efforts in digitalization and especially to develop AI tools in support of the public administration. One of the major private investments in the development of AI – by Wallenberg Autonomous Systems and Software Programme (WASP) – was extended with special research application calls and a graduate school focusing on AI from the perspective of social sciences and the humanities. The program is led by Virginia Dignum at the Department for Computing Science at Umeå University.
Besides these initiatives in research and development, there have been developments in terms of regulation and oversight of automated decision-making in the public sector. Among other things, the National Audit Office began large-scale oversight of automated decision-making within public agencies at the national level. Furthermore, we have seen several litigation cases related to automated decisions for social benefit applications in the municipality of Trelleborg (the so-called Trelleborg model) as well as the first GDPR ruling in Sweden concerning a face recognition system in a gymnasium.
A catalog of ADM cases
To understand how the public sector is using ADM, we conducted a systematic mapping exercise. In November 2019, we sent freedom of information requests to 321 government agencies and supporting units to find out if they:
a) Worked with ADM
b) And, if they did, in which areas they use
By the end of November, we had received 213 responses. 84 agencies did not respond within the timeframe and 24 could not be reached. Out of the 213 agencies who replied, only 15 said that they relied on ADM (often in the form of robotic process automation (RPA) or automated process flows).
The replies showed that there is some confusion as to what the term ADM encompasses. We received several questions concerning the definition of ADM, as well as longer elaborations of how ADM is understood at particular institutions. Several respondents only referred to RPA as a form of ADM, while others also included algorithms that are part of external infrastructure that they rely on for their work (e.g. social media tools that are used for external communication).
The agencies that are at the forefront of ADM are the Pensionsmyndigheten (Swedish Pensions Agency), Försäkringskassan (Swedish Social Insurance Agency) and Skatteverket (the Tax Agency). Another important agency working with proactive open data initiatives is the Lantmäteriet (Mapping, Cadastral and Land Registration Authority) that has, among other things, automated decisions related to real estate mortgages.
One agency that has automated many processes that directly affect applicants is Försäkringskassan (Swedish Social Insurance Agency). This agency, together with the Swedish Association of Local Authorities and Regions, owns and develops the SSTBK database. SSTBK is a comprehensive platform that combines databases from different public agencies to allow for decisions on social benefits (agencies included are Swedish Public Employment Services, Swedish Board of Student Finance, Swedish Federation of Unemployment Insurance Funds, Swedish Social Insurance Agency, Swedish Tax Agency).
Public Employment Services
In the following section, we will focus more closely on the Public Employment Services for two reasons. Firstly, the agency is actively working and experimenting with ADM and machine learning. Secondly, the organization is currently undergoing a major restructuring including substantial budget cuts that have contributed to the accelerated development of technological solutions for automation.
Budget cuts of around 2.9 billion SEK inled to 1700 employees being laid off and the closure of offices, especially those outside the big cities where many vulnerable communities reside. The agency aims to direct most of its clients to remote services and only provide assistance at local offices for a minority of cases. Prior to the budget cuts, the agency had plans to increasingly automate decision processes. The cuts have led to an increased need to implement automation to handle the workload. However, this has to be done with decreasing resources at the same time as the agency prepares for an anticipated recession and a subsequent rise in the number of unemployed people.
Currently, the Public Employment Services are working with automation in three areas:
- activity reports
- financial checks of companies applying for support when employing people who have been unemployed for a long time (företagsstöd)
At the moment, the first contact unemployed people have with the Public Employment Services is exclusively digital and self-service. Job seekers have to self-register online and are then booked for a telephone or video call within a month to evaluate their prospects of returning to the work force (arbetsmarknadspolitisk bedöming).
If clients are eligible for unemployment benefits and enter an educational or training program, they have to report on their job search and program activities to the Public Employment Services on a monthly basis. The registration, evaluation, and decision on whether to provide continued support are automated. Finally, ADM is also used to perform financial checks on companies seeking financial support to help them employ the long-term unemployed. The eligibility of a company to receive this support is automatically rated: green = eligible for support, yellow = needs an additional manual check, and red = not eligible). According to an interviewee we spoke with, there are limited possibilities for caseworkers to challenge the automated decision in this specific area.
Adult Education and Economic Support – Nacka Municipality
Nacka municipality is part of Stockholm county, and it has developed and implemented a so-called robot employee called Yasmine. The robot is part of an RPA service platform that was provided by the Norwegian company Basefarm. So far, the administration of Nacka has automated three processes. Two of these are in the so-called Establishment Unit (Etablering), and the other is in the Work and Business Unit. The Establishment Unit is responsible for economic support, unaccompanied children and minors (refugees). The Work and Business Unit administers adult education, societal orientation, and labor market initiatives (Nacka Kommun 2018).
The current automation project was modelled on a similar automation project in Trelleborg municipality (see previous report), where decisions on applications for social benefits have been fully automated. Trelleborg is the first municipality to work with fully automated decision-making on social benefit applications.
Nacka has developed a centralized approach that allows all units to start automation projects. Different units and departments are supported in developing a detailed project plan and workflow description for implementing ADM (Hertin & Lindberg, 2019). Units interested in automating can apply and implement a pilot project and receive support from the central administration unit to do so.
This centralized approach is supposed to increase the degree of automation of the whole administration of the municipality while using a common approach across different units and work processes. The processes that are currently automated are related to adult education (15 rules developed for RPA) and decisions on economic support (200 rules developed for RPA). Basefarm provided the platform, and the tool, which was designed by a company part-owned by Google Alphabet (CapitalG investment) is called UiPath (UiPath 2020).
Nectarine – AI in elderly care
Nectarine Health (a company previously called Noomi and Ai-floo) has developed a wristband for the elderly. It is fitted with multiple sensors that monitor eating habits, movement, sleep patterns, and can detect if the wearer trips or falls (Nanalyze 2018)(Svensknarings 2018). Based on data from the wristband, caregivers are automatically alerted in case of an emergency. In 2016, Nectarine Health started collaborating with Aleris, a major private healthcare provider in Sweden, when the wristbands first came into use.
Apart from registering movements, the wristbands also monitor other health-related data mainly to organize elderly care more efficiently. This example is included here to illustrate that Sweden is uniquely placed in terms of collecting and sharing patient data in different public and private registries that are increasingly harvested by international and transnational private companies (Gustafsson & Röstlund, 2019).
Swedish microchipping movement
The Swedish microchipping movement, which took off in 2015, has been in the international news again (Läsker 2019)(Schwartz 2019). It is estimated that around 5000 people in Sweden currently have RFID technology-based microchip implants that allow them to automatically open doors and use photocopiers.
On several occasions, microchipping has become a popular feature at parties. For example, one of these microchipping events was arranged during Almedalen, a high-profile political week held during the summer of 2017, by the Swedish state-owned company Statens Järnvägar (SJ) (Petersén, 2019).
School Placement Algorithm
In spring 2020, a report by the state (statens offentlig utredning) into school segregation was made public (Statens Offentlig Utredning 2020). Among other things, the report suggests implementing a more complex algorithm for the placement of school children. At the moment, school selection is based solely on the relative proximity of the school.
In Gothenburg, one case illustrates the failure of the automated system as the algorithm only considered the straight line distance to a school, and not the geography, or the actual time it takes to get to the school building by foot or car (Dagens Nyheter 2020). After school placement decisions are made in the spring, reports often emerge of parents who are unhappy with the automated decisions. In 2018, parents in Uppsala, a city north of Stock-holm, developed a placement algorithm that they believed would make the assignment fairer. However, the city board in charge never considered the algorithm (Heimer 2018).
Swedish Police Piloting Face Recognition Software
As part of a broadly discussed special operation to fight organized crime (operation Rime Ice), the Swedish police implemented several face recognition pilot projects including searching through visual material such as pictures from crime scenes and CCTV footage to identify reoccurring people. The police did not inform the Swedish Data Protection Authority about the pilot testing. Last year the authority ap-proved that the police could continue matching face shots with an offender register (signalementregister). However, their application for testing face recognition at Skavsta air-port was dismissed in 2019. In 2020, the government submitted a revised application to implement a pilot project from January 2021 (Munther 2020). At the moment, the Swedish Data Inspection Authority is reviewing the police’s use of the American face recognition app Clearview.
Policy, oversight and debate
Report by the Nordic Council of Ministers
In 2019, the Nordic Council of Ministers published a report about the use of artificial intelligence at the municipal level across the Nordic countries (Andreasson & Stende, 2019). The authors focused explicitly on AI that they define as autonomous systems that are apparently intelligent. The report does not explicitly define ADM, but refers to a number of tasks that are or can potentially be automated with the help of AI. These systems are based on data analysis and can solve different tasks with a certain degree of independence. AI, according to their definition, is based on algorithms made up of mathematical formulas or rules that define how AI should act. The report focuses, in particular, on machine learning as a specific form of AI, based on algorithms, that are self-learning, self-adjusting, and self-improving over time, and that can derive rules, and detect patterns in large datasets.
The report focused on two municipalities in each of the five Nordic countries (Denmark, Finland, Iceland, Norway, and Sweden) and mapped their work on AI and ADM. One of the main takeaways from the report is that, while neither AI nor ADM are used very broadly in any of the Nordic countries, local municipalities are running several different pilot studies. These studies tend to focus on chatbots, which are increasingly common at the municipal level.
The report emphasizes that the use of AI and ADM might have a large impact on trust in public institutions if it is not implemented in a sustainable and sensible manner. The Nordic countries have a long history of strong social trust, and trust in public institutions, and the authors of the report provide three general suggestions. Firstly, municipalities in the Nordic countries should continuously exchange experience with AI implementation projects. Secondly, there should be shared ethical guidelines across the Nordic countries, and thirdly, they should come together to invest in research and development projects to concentrate re-sources and spend funding efficiently.
Report by the Union for Professionals
In 2019, the Union for Professionals (Akademikerförbundet SSR) published a report written by Lupita Svensson, a senior lecturer at the School of Social Work at Lund University, on ADM within the Swedish social service system (Sven-sson, 2019). Svensson evaluated and mapped ADM across all municipalities with the help of RPA. Her report showed that only 16 out of a total of 290 municipalities have implemented RPA in their administration of social benefits. However, only one – Trelleborg municipality – has implemented fully automated decisions, while the other 15 municipalities use RPA for decision support. Besides mapping the extent to which Swedish municipalities use automated processes in social services, Svensson also conducted interviews with politicians, project leaders, and case workers across municipalities. She came to the conclusion that the technological shift that the Swedish social services are currently going through can be compared to computerization in the 1980s. However, there are important differences. The policy for social services during the 1980s emphasized the importance of trust, whereas current policies, and the technological solutions they utilize, emphasize control. At the same time, caseworkers argue that they have more time to actually do social work, although they need to renegotiate what their basic tasks encompass, besides a strong focus on administration and documentation. This new wave of “deep” digitalization that goes beyond earlier e-governance initiatives is still ongoing and is characterized by contradictions. In the broader context, the need for support including social benefits is increasing, while resources are systematically shrinking. Automation and digitalization are often motivated with efficiency and cost-saving arguments how-ever, the necessary infrastructure is often cost-intensive and needs continuous maintenance and these problems are often downplayed in the discussion. Svensson’s findings point to a fundamental shift in how social services are organized and delivered to citizens.
Report by DIGG – Agency for Digital Government
In Spring 2020, the Swedish Agency for Digital Government (founded in 2018) published a report for the government entitled To Support the Public Administration in Using AI (DIGG, 2020). The report maps the pre-conditions for the use of AI in public administration, the main challenges, and develops a set of recommendations to increase the use of AI in public administration. The starting point is that the costs and resource requirements for both the public and welfare sector will increase considerably over the next decade. The report states that AI offers huge economic potential that, until now, has not been fully exploited. AI is connected with the possibility to increase, and make more equal, access to health services, improve education, and implement more just and efficient decision-making in public administration. The report points out two particular challenges. Firstly, questions of data security and secondly, ethical considerations. In order to support the implementation of AI in public administration, the report makes the following suggestions:
- Create a center for expertise in AI
- Establish a platform for coordination, co-development, and innovation
- Develop an AI guide for public administration
- Establish the juridical foundation to allow for test-bed activities
- Develop professional and role-based education for AI
- Develop a national data strategy for public administration
First GDPR fine in Sweden for the use of face recognition at a public school
In 2018, a public high school in Skellefteå, in Västerbotten County, used face recognition to register student attendance (Lindström 2019). One class took part in a three-week pilot of a system implemented with Tieto (IT Pedagogen 2019), a Nordic IT software and service company with headquarters in Finland. However, the Data Protection Authority (Datainspektionen, 2019) issued a fine of 20,000 euros (200,000 SEK) to Skellefteå municipality – which is the first recorded GDPR fine in Sweden – for unlawful use of face recognition software.
The Data Protection Authority ruled that GDPR had been violated in several ways (Heinje 2019). Firstly, the use of face recognition technology was considered excessively intrusive and, in that sense, it violated the principle of data minimization (article 5(1)(c)). Secondly, face recognition falls under ‘special categories of data’ (article 9) that require specific consent by the data subject. Even though the students gave their informed consent, their position as students (and minors) meant that the decision to participate was not a real choice. Thirdly, the school did not document a Data Protection Impact Assessment (DPIA) and hence violated article 35.
The relatively modest fine was imposed because of the brief duration of the pilot scheme, and the fact that only 22 students were involved. Skellefteå has challenged the decision by arguing that participation in the scheme was voluntary and that students and their parents who opted into the pilot gave their full consent to the gathering of data (Lindström 2019).
Trelleborg municipality reported to Parliamentary Ombudsperson
Our first report for Automating Society highlighted the automation project of Trelleborg municipality that fully automated decisions on social benefit applications and has worked on this particular ADM project since 2017. In 2019, Simon Vinge of the Union for Professionals (Akademiker-förbundet SSR) reported the municipality of Trelleborg to the Parliamentary Ombudsperson (Justitieombudsman-nen JO). His complaint was that the authorities failed to respond to a Freedom of Information (FOI) request related to the use of RPA to make automated decisions. As of May 2020, they have not heard back from the Parliamentary Ombudsmen.
Similarly, the journalist Freddi Ramel started a discussion about transparency related to the ADM system used in Trelleborg. After several failed attempts to get access to the source code from Trelleborg municipality, and after reaching out to the Danish company responsible for the coding, he took legal action. In contrast to Simon Vinge, Ramel sub-mitted an appeal to the Administrative Court of Appeal, by arguing that the source code of the software falls under the Swedish Offentlighetsprincipen (the principle of public access to official records). Following his appeal, the court ruled that the source code had to be made publicly accessible and fully included in the principle of public access (Da-gens Samhälle 2020).
After this ruling was made public, another journalist at the financial newspaper, Dagens Industri, submitted an FOI request to view the source code. Trelleborg municipality for-warded the request to the Danish software company, who then shared the source code with the journalists. However, the code was not checked beforehand, and sensitive data, including personal identification numbers (similar to social security numbers), names, and information about home care for the elderly, was also shared with the journalists (Dagens Samhälle 2020).
Mapping and review by the National Audit Office
In 2019, the National Audit Office (Riksrevisionen) conducted an initial mapping of ADM currently in use by public agencies. Based on the results, it decided to undertake a larger review of ADM used by the public sector, and that has a direct impact on citizens. Their report will be published in October 2020.
There is an ongoing and vivid debate about ADM, which is often framed as AI in the public sector, and there have been numerous events, panel discussions, reports, etc. At the same time, there are not too many instances of fully automated decision-making. Having said that, public institutions continue to test the water and navigate grey legal areas. The dominant forms of ADM are those made up of RPA and based on solutions provided by companies like Blue Prism and UIPath. Automation and pilot projects are often implemented in collaboration with external consulting firms such as PricewaterhouseCoopers or Boston Consulting Group. The Swedish Association of Local Authorities and Regions coordinated the introduction of ADM. Regarding automation projects, private companies are increasingly becoming part of, and are responsible for, critical social infrastructure.