In 2018, the Polish government initiated a public debate on artificial intelligence technologies by preparing a draft AI strategy through a participatory process that involved a broad range of experts and stakeholders. Throughout the debate, we observed limited interest from stakeholders to discuss ADM specifically, and, more broadly, the societal impact of algorithms – although the ethics of AI were nominally included in the process. The debate among stakeholders, initiated by the government, has focused on defining Poland’s strategy for supplying AI solutions. Key topics concern: support for Polish entrepreneurs, provision of necessary digital talent, and the capacity of Polish research institutions. Some attention has been paid to regulatory issues, in particular those related to the availability of data for AI solutions.
At the same time, for the purposes of this report, we looked at institutions employing ADM systems but we found a very limited number of cases. One prominent example stood out: an automated system for assigning court cases to judges. This is a case that received public attention by virtue of being part of a broader, controversial (political) reform of the judicial system. Yet even in this case, the lack of transparency around the project made it difficult to establish the characteristics of the system and its real impact. There were a few other cases, but they received little attention from the media or wider society and caused limited controversy. One example – an important ADM development case study – is the system used to profile unemployed people. However, it did not garner any significant public attention.
At the end of 2018, the government unveiled its draft AI strategy, which was scheduled for finalization the following year. 2019 was to be the year in which the Polish government would define the final shape of its AI strategy. In Poland, the public administration is seen as a key actor that can help shape the national approach to AI. This is due to the high level of public funding available to core parts of the IT sector. Additionally, academic researchers – whose work is predominantly funded from public sources – are also seen as an important part of the Polish AI ecosystem.
The third key group of actors – startups and other IT companies – have not been very active in these public conversations, despite the fact that other stakeholders assign them key roles. These private companies are largely software houses that provide AI-based solutions directly to their clients, largely foreign ones. As a result, they have little interest in participating in national debates about AI and ADM.
Throughout 2019, it became clear that it would not be possible to finalize the national AI strategy quickly. A final draft of the strategy was made available for public consultation in late August. However, the whole process had yet to be finalized by the end of 2019. During 2019, no significant ADM-related initiatives were announced in the public sector and a very limited number of AI-based solutions were introduced by commercial actors. The ADM systems that directly impact Polish citizens can mostly be found in the financial sector and on various online platforms – yet these receive very limited public attention or oversight, mostly due to their global scope. Finally, 2019 has been the year in which ADM as a public policy issue was introduced into the public debate, largely due to the activity of several non-profit organizations.
The broad issue of “artificial intelligence” remains a popular topic in the media – for example, the development of “deep fake” technologies has received widespread attention in the mainstream media. However, media coverage rarely concerns specific ADM cases. Mainstream media outlets generally offer limited coverage of “tech” topics and focus on subjects that are seen by media titles as controversial or possibly interesting to the general public. For this reason, there is very limited coverage of ongoing developments that concern the development of AI or ADM solutions in Poland. In this context, it should be noted that in early 2019 the National Information Processing Institute (OPI) launched a dedicated service covering topics related to AI technologies.
However, the most controversial ADM-related debate focused on “content upload filters” used by Internet platforms, which was a highly contentious issue in the Euro-pean copyright reform debate. Poland was one of the countries, in which this debate was the most heated. In late March 2019, the ruling party (Prawo i Sprawiedliwość) declared that protecting users’ freedom online through an appropriate implementation of the new Directive would be one of its electoral promises in the upcoming elections to the European Parliament. In practical terms, this declaration meant that Poland will strive not to implement the rules that enforce the controversial content upload filters. The decision was an attempt to build support among right-wing voters, seen by the party as being largely opposed to the new content filtering provisions. In May 2019, Poland challenged the content filtering provisions at the European Court of Justice. While the term ADM was never used during this policy debate, this challenge should be seen as an important case where the societal impact of ADM systems came up for public debate.
A catalog of ADM cases
There were no major developments regarding the use of ADM in 2019. Having said that, significant change can be seen when it comes to policy debate and judicial decisions although not when it comes to implementation. The reasons for such a state of play are multifaceted. Firstly, Polish administration, which is perceived as a key actor when it comes to the development of AI and ADM solutions, does not treat the development of ADM solutions as a priority. Up until now, the development of an e-administration infrastructure – which is the task of the Ministry of Digital Affairs – has focused on systems that do not employ ADM mechanisms. Secondly, Poland is still a country where the supply of ADM solutions remains higher than demand. Whereas Polish IT companies provide ADM solutions across the world, there is no need to implement them locally – whether it be in the public or the private sector. Generally, the demand for AI services in Poland is limited and this creates a natural tendency to seek out clients abroad. In Poland, 73% of companies receive at least some of their revenue from clients abroad and 33% receive most of their revenue from foreign clients.
According to the Map of the Polish AI 2019 report, published by Digital Poland Foundation, the commercial ADM solutions that are most frequently supplied by Polish AI companies are analytics, big data, and business intelligence, together with sales, marketing, and advertising (one of the examples, Mr. Wolf, is presented below). In addition, companies often provide services in areas such as; financial services and insurance, the Internet of things, and industry 4.0. This is probably due to the relative competitiveness and openness to innovation in the Polish financial and industrial sectors. However, it has to be underlined that AI solutions used by companies are usually related to automatization, not necessarily to taking any decisions. The more general overview of such companies is presented in the report Iloraz Sztucznej Inteligencji 2 (only available in Polish). ADM solutions are employed to a limited extent by the Polish business sector, however, growth in this regard can still be seen. As mentioned above, Polish companies develop AI solutions largely for foreign markets and thus have limited influence upon the Polish ecosystem.
Detection of bank accounts being used for illegal activity
STIR – System Teleinformatyczny Izby Rozliczeniowej (Clearance Chamber ICT System) is a new tool that gathers monetary information from banks, Cooperative Savings, and Credit Unions. The aim of the system is to examine financial activity and to discover potential illegal activity. The system is operated by Krajowa Izba Rozliczeniowa (State Clearance Chamber – a key entity of the Polish payment system infrastructure, which renders complex clearing services and provides solutions for the banking and payment sectors). The ADM tool in this system provides suggestions to tax authorities by assigning risk indicators to accounts. For example, if there is suspicion that an offense has been committed related to a certain account, then (at the request of the tax authorities) the bank can block the account for 72 hours (which can be extended).
Legal regulations regarding STIR have been in force since January 13, 2018, when amendments introduced to the Act of August 29, 1997 – on Tax Ordinance – came into force.
The algorithm used by STIR is not publicly available, nor transparent. Moreover, the law introducing STIR states that disclosing or using algorithms or risk indicators, without being entitled to do so, is an offense. A person who discloses algorithms can be imprisoned for up to five years (if the act was unintentional, then a fine can be given).
The risk indicators, which are a result of an assessment to see if money has potentially been used for illegal activities, is determined by the algorithms developed by the Clearance Chamber. This risk indicator takes into account the best practices of the banking sector in the field of counteracting financial and tax offenses, and it is a key factor when deciding whether or not to block an account. These criteria might be as follows (according to Tax Ordinance):
- economics (based on the assessment of overall economic activities of an entity, especially taking into consideration any transactions unjustified by the nature of the business),
- geography (consisting of transactions with entities from countries in which there is a high risk of tax fraud),
- subject-specific nature of certain entities (conducting high-risk business activities when it comes to the possibility of tax extortion),
- behavior (any unusual behavior of the entity) and
- connections (the existence of links between the entity and entities that are at risk of participating in activities related to tax fraud).
The Clearance Chamber can assign the task of maintaining or modifying the STIR system to an external contractor. Such delegation is made based on a civil law contract, but the public supervision of the process whereby an entrepreneur is selected is not sufficient. This is due to the limited application of public procurement rules (the State Clearance Chamber is a stock company). This might lead to insufficient transparency and oversight of this ADM system. In reply to a freedom of information request by the ePaństwo Foundation for more information about this particular entrepreneur, and the contract itself, the Ministry of Finance refused to disclose the information on the basis that the State Clearance Chamber is a private entity and that its operations cannot be revealed under FOI laws.
In December 2018, the Administrative Court in Warsaw issued a precedent judgment in the STIR case. The court stated that in the case of extending the account lock by three months, there is no need for an evidence proceeding to be conducted. In such a case, it is enough for a public officer to analyze the flows of money and to assume that there is a risk of using a certain bank account for a tax scam.
Smile-monitoring of employees and consumers by PKO bank
A rare example of media coverage of ADM occurred in September 2019, when the Gazeta Wyborcza daily wrote about a test pilot of Quantum CX, an ADM system that has at its heart a machine vision algorithm which is able to distinguish the smiles on people’s faces. As part of the pilot, the system was installed in several branches of the PKO BP bank. The system tracks smiles of both employees and clients. The former can exchange smiles for prizes, while on the basis of the number of smiles of the latter, the company pays funds to a charity.
The creators of the system present it as a modern, AI-based approach to employee motivation and corporate charity programs. Participation in the pilot is voluntary and employees are able to independently regulate their own privacy settings in a specially designed application. Using the highest-level privacy settings means that the manager cannot check the results of a particular employee, but only the sum of smiles in a given bank branch.
After the initial article, the system was criticized by lawyers working on digital rights and labor law issues. The issue was addressed by the Polish Ombudsperson (Rzecznik Praw Obywatelskich), who questioned the legality of the system on the basis of both labor law and GDPR compliance. The Ombudsperson has sent a formal request to the Chief Labor Inspector to investigate the case.
At the same time, the system has received a positive response from potential clients. Another pilot has been initiated by the office of the city of Sopot. In addition, several companies have expressed an interest in the system as well.
The case of Quantum CX clearly demonstrates the varied responses to ADM systems, and we will have to wait and see how the case will further develop in 2020 and whether any further evaluation of this system will be conducted.
The ADM system for profiling the unemployed is scrapped
As presented in last year’s report, in May 2014, the Ministry of Labor and Social Policy introduced a simple ADM system that profiles unemployed people and assigns them three categories that determine the type of assistance they can obtain from local labor office. Panoptykon Foundation, and other NGOs critical of the system, have been arguing that the questionnaire used to evaluate the situation of unemployed people, and the system that makes decisions based on it, is discriminatory, lacks transparency, and infringes data protection rights. Once the system makes a decision based on the data, the labor office employee can change the profile selection before approving the decision and ending the process. Yet according to official data, employees modify the system’s selection in less than 1% of cases. This shows that, even if ADM systems are only being used to offer suggestions to humans, they greatly influence the final decision.
The issue was finally solved in 2018, when the Constitutional Tribunal decided that the system needs to be better secured in a legislative act – although its main objection was rather formal: the scope of data used by the profiling tool should have been set out in the legal act adopted by parliament and not decided by the government. As a result, the government decided to end its experiment with profiling the unemployed, and the system was finally scrapped in December 2019.
The case of this ADM system shows that assessing the impact of even a simple ADM system is a resource and time-intensive effort. The case also proves that – while the public administration lacks the necessary procedures, institutions, and skilled staff – the role of non-profit watchdogs who monitor ADM development is crucial.
Mr. Wolf is a commercial tool that helps to automate text communication in customer service by automatic classification of notifications, interpretation of their content, and by providing automated answers to repetitive questions. The system makes automated decisions related to consumer service communication at various levels, starting from classification to content decisions. After being sent to the system, notifications are categorized by algorithms. Mr. Wolf’s task is to identify what kind of information is contained in the notification: is it a bug report, parcel order or a password reset? It then assigns the case to the appropriate category. In the event that the question cannot be categorized, it redirects the task to a human and then the process of answering begins. After receiving the information, the software provides answers or transfers the matter to the employee. What is important is that the cases that are resolved by employees teach the system how to solve more difficult cases – which make ADM systems more effective and failure-proof. Mr. Wolf is an example of an ADM tool that makes repetitive actions less burdensome and time-consuming. At the same time, from the way the software has been programmed, it accepts the fact that not all cases can be solved automatically and that there is a need for humans in the process.
Policy, oversight and debate
Artificial Intelligence Development Policy for 2019-2027 and other governmental actions
The Polish government has been working on the issue of Artificial Intelligence since June 2018, when Jarosław Gowin, Deputy Prime Minister and Minister of Science and Higher Education, declared that Poland will create its own Artificial Intelligence strategy. As of July 2020, the final version of the national AI strategy still hasn’t been published.
The work on a national AI strategy, and programs emerging from this strategy, have been relatively slow. Efforts related to AI development were initiated by two different Ministries: The Ministry of Entrepreneurship and Technology, and the Ministry of Digital Affairs. These two agencies have coordinated their efforts to a small extent. The former is focused on industrial policy related to the “Industry 4.0” model, while the latter is developing educational programs aimed at increasing the supply of skilled digital talent in the IT sector.
On February 26th, 2019, the Minister of Digitization, Minister of Entrepreneurship and Technology, Minister of Science and Higher Education, and the Minister of Investment and Development all signed a memorandum on the development of Artificial Intelligence in Poland which aimed to provide a strategic and cross-sectoral framework for AI development. The Ministries agreed to create an AI ecosystem, to coordinate activities aimed at providing interdisciplinary education and research on AI, to provide citizens with skills related to data science, and to cooperate in order to create a roadmap to get there.
In late August 2019, the Ministry of Digital Affairs published the much-awaited final draft of the Artificial Intelligence Development Policy for 2019-2027 for public consultation. At the time of writing this report neither the results of the consultation nor the final version of the policy document have been made public.
The key goal of this document is to provide Poland with an important place in the global economy, which is already being shaped – and will be further shaped in the near future – by AI technologies and applications. In the opinion of its authors, the use of AI by Polish companies is necessary to build a strong internal market for AI solutions. As written in the document, the mission of Poland’s strategic policy in the field of AI is to support AI science, research and development for the growth of innovation, and productivity of the knowledge-based economy, as well as to support citizens in the process of transformation in the work environment, and the improvement of competences while taking into account the protection of human dignity, and ensuring conditions for fair competition. So far, AI development in Poland has led to the creation of a supply strategy in which Poland is a country providing AI-based solutions that are then used in other parts of the world. Creating a demand strategy is necessary if AI technology is to benefit Polish society. Such ambition is visible in the draft policy.
Even if the term ADM is not mentioned in the draft of the document, the policy refers in many places to decisions taken by AI, which constitute part of general ADM mechanisms. The authors of the policy acknowledge that using AI in ADM is inevitable, but that these systems shall be implemented with respect to certain standards, such as transparency (including how algorithms work). Moreover, ADM development shall be in accordance with international standards, legal framework, and ethical considerations. The documents also put legal challenges related to ADM in the spotlight, such as the automatic conclusion of civil contracts. There are also certain proposals to help regulate ADM, such as a system of norms, certification or data management standards – although a single perfect solution has not been identified.
The authors of the document summarize the framework by concluding that, in order to strengthen AI development in Poland, coordination is necessary in the following areas: construction of a 5G network; development of digital competences at all levels of education; support for the possibility of creating inter-university networks to create consortia dealing with specific research problems; promote the latest technological solutions among entrepreneurs to raise the level of development and establish a dialogue with other EU economies; support and promote joint entrepreneurial solutions (e.g., joint R&D, exchange of experiences, cluster solutions); support cooperation between academic centers and business entities, support projects in the field of cybersecurity; support the creation of an API standard for access to industrial data and open interoperability standards.
The right to explain credit decisons
Up until recently, in Poland and other European Union countries, only entrepreneurs had the right to ask a bank to explain a decision to decline credit. Poland is the first EU country to extend this right to individual customers and to clarify what information banks must provide to their customers – including cases where ADM was used to calculate a credit decision. At the end of April 2019, an Act of 21 February 2019 amending other acts (including banking law and the Consumer Credit Act) in connection with ensuring the application of the General Data Protection Regulation came into force.
According to the new law, everyone has the right to clarify their credit assessment if:
- they are applying for any kind of loan. It does not matter if the decision was negative or positive or whether the application is for a mortgage or just for the ability to shop in installments.
- they are applying for a consumer loan at a loan institution (companies such as Provident, Vivus, and Stork operate in the Polish market), and the final decision was negative.
If a credit or loan decision was made automatically (without human intervention), citizens have the additional right to have the basis for that decision explained to them. This permission is independent of whether the automatic decision was positive or negative. In this case, the customer also has the right to express their own position. If someone does not agree with the ADM decision, their application can be reviewed, and the resulting decision will be made by a human (e.g., a bank analyst). This procedure applies both at banks and at lending institutions.
The clarification can be requested as soon as the bank, or loan institution, issues a credit decision. The additional right to explain the grounds for an automatic decision can also be exercised as soon as the result of the decision is known. In practice, this occurs immediately after submitting the application – automatic decisions are made as soon as all the required data has been entered into a computer.
Civil society and academia
While the general public and policy debate in Poland focuses on AI and its commercial potential, civil society actors are trying to shift the narrative more towards ADM-related issues. According to NGOs (Centrum Cyfrowe Foundation, Panoptykon Foundation, Klub Jagielloński, and others), discussion about ADM systems – as opposed to discussion about AI – allows for a better understanding of the impact of new technologies on the individual and society. What is worth underlining is the fact that this shift is not purely dominated by NGOs, but also allows the space for dialogue with various other actors, especially administration and business.
AlgoPoland. Automated Decision Making in service of the society
The AlgoPoland Automated Decision Making in service of the society report is the result of cooperation between the Centrum Cyfrowe Foundation (Polish digital think-and-do tank, of which the authors of this chapter are members) and Klub Jagielloński. The key assumption of the report’s authors is that Poland has so far implemented fewer ADM systems than other European states and developed economies. To encourage sustained development of ADM systems, Poland can take advantage of the benefits of its delayed economic growth. While implementing these systems, Poland is able to learn from the mistakes made by other countries which have pioneered the way but have often made faulty attempts at implementation. A rational strategy for the implementation of ADM systems – one based on clear principles aimed at serving the public interest – can be of great benefit to the Polish economy, society, and public administration.
The AlgoPoland report consists of three parts: presentation of various applications of ADM systems around the world, and specifically in Europe and Poland, in the following key areas of social life: predictive policing, job market, social welfare, and healthcare, education, access to information and culture, finance, the judiciary system, and finally, the challenges and opportunities related to ADM and recommendations.
The recommendations from the report are as follows:
- To build socio-economic trust regarding ADM systems, it is vital to proceed within the quadruple helix model which combines academia, industry, public administration, and civil society.
- ADM systems must be regulated. Due to their considerable impact on individuals and societies, we need to cooperate to develop regulatory measures which will ensure that ADM systems serve the joint interests of the economy and society.
- ADM systems are only as good as the data which they process. High quality and, most importantly, the availability of datasets are necessary for avoiding biased and corrupt results of using ADM systems.
- ADM systems must be based on a transparent and explainable mechanism.
- The key to sustainable development of ADM is public awareness: understanding of the mechanism and results of the application of such systems.
AlGOVrithms. State of play report
In May 2019, another digital NGO – ePaństwo Foundation, in cooperation with partners from the Czech Republic, Georgia, Slovakia, Hungary, and Serbia, published a report called AlGOVrithms. State of play report. The report identifies examples of algorithms used in the public administration systems of the previously mentioned countries. The report describes how they work, verifies whether they were introduced through legal regulations, checks how open and transparent they are, and sees if there is any system in place that secures citizens’ rights against potential abuse as a result of algorithms. While working on the report, researchers mainly used three methods to collect the required information: desk research, freedom of information requests, and interviews with recognized experts and decision-makers.
The authors did not outline the existing overall state policy regarding the implementation of alGOVrithms in any of the countries participating in the research. Moreover, they did not find any examples of the existence of a legal framework comprehensively describing the rights and obligations of the states and citizens regarding ADM. The legal documents that do exist refer to some aspects of examples of alGOVrithms, such as systems to allocate judges to specific court cases. The authors proved that algorithms used in ADM software are not subject to transparency and that access to the algorithms or the source code is not possible. In addition, the authors did not find any cases where a single institution oversees, or even possesses comprehensive knowledge of, which ADM systems exist in each country. Apart from the example in the Serbian system, where judges are allocated to cases, and where the donor (EU) has audited the system, no external and independent audits are in place to monitor the accuracy and fairness of the operation of algorithms. The report shows that there is a long way to go in the region before a sustainable and human rights-proof policy framework for ADM development is created.
Digital Sustainability Forum
The Digital Sustainability Forum (DSF) is a joint initiative between the Centrum Cyfrowe Foundation, Digital Poland Foundation, and Microsoft. The goal of the initiative is to develop recommendations for decision-makers who are responsible for building strategies, policies, and regulating emerging technology. These recommendations are the result of cooperation between representatives of science, business, non-governmental organizations, and public administration. The cooperation of all participants of this innovative ecosystem has resulted in the development of pragmatic recommendations which take into account the reality of a constantly changing world. It is also a tangible example of a project using a quadruple helix model which combines academia, industry, public administration, and civil society.
Up until now, the Digital Sustainability Forum was the place to discuss two issues: AI and cybersecurity. The AI discussion addressed the questions of how to build digitally sustainable policies in the AI sector, how to ensure that the opportunities provided by the development of AI technology overcome social risks and challenges, and how to use Poland‘s digital transformation to achieve its sustainable development goals. The issue of safeguards that need to be implemented when it comes to ADM systems that use AI was also discussed. The scope of the forum shows that the issue of ADM (and other systems) need to be addressed from various perspectives. Such an approach creates room to help answer the challenges of ADM from both a societal and a business perspective – the inclusion of diverse voices will be crucial in this process.
In 2019, the public debate on AI, which was initiated when the government began work on the national AI strategy, continued with a significant number of events and reports covering the issue. AI is now a prominent topic of discussion among stakeholders interested in the development and regulation of digital technologies in Poland.
Within this broad context, the specific concept of ADM has mostly been promoted by several non-profit organizations, but this has received limited recognition from the government. The overall debate remains largely focused on AI and its economic potential, whereas the public administration has been paying a degree of attention to issues such as AI ethics and data governance.
In 2019, we did not observe any significant new deployments of ADM systems in Poland. In particular, it was noted that the public sector did not deploy any such solutions throughout the year. Several ADM systems, which we have identified in this report, have very limited scope and have not sparked any significant debate on ADM systems in Poland.
Poland has a strong IT sector and many companies are working on services based on AI solutions. These companies are commonly seen as the strongest asset of the Polish AI ecosystem, but they usually develop bespoke systems for foreign and not domestic clients, and for this reason they have a very limited impact on the Polish AI ecosystem and surrounding public debate.