CHAPTER
3.3. Acquisition of Automated Decision Systems for High-Risk Applications
12114.
This chapter shall be known and may be cited as the Automated Decision Systems Accountability Act of 2021. Act.12114.5.
It is the intent of the Legislature that agencies of the state use an acquisition method that minimizes the risk of adverse and discriminatory impacts resulting from the design and application of automated decision systems.12115.
For purposes of this chapter, the following shall apply:(a) (1) “Automated decision system” means a computational process, including one derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or
recommendation, or other simplified output that is used to support or replace human decisionmaking and materially impacts natural persons.
(2) “Automated decision system” does not include a tool that does not automate, support, or replace human decisionmaking processes, including, but not limited to, a junk email filter, firewall, antivirus software, calculator, spreadsheet, database, data set, or other compilation of data.
(b) “High-risk application” means a use of an automated decision system for which any of the following apply:
(1) Poses a significant risk to the privacy
or security of personal information or has the potential
is likely to result in inaccurate, unfair, biased, or discriminatory decisions impacting natural persons, taking into account the novelty of the technology used and the nature, scope, context, and purpose of the automated decision system.
(2) Affects the legal rights, health and well-being, or economic, property, or employment interests of a natural person, or otherwise significantly impacts a natural person. person.
(3) Involves the personal information of a significant number of individuals with regard to race, color, national origin, political opinions,
religion, trade union membership, genetic data, biometric data, health, gender, gender identity, sexuality, sexual orientation, criminal record, or any other characteristic identified in the Unruh Civil Rights Act (Section 51 of the Civil Code).
(4) Meets any other criteria established by the Department of
Technology in regulations issued pursuant to Section 12117.
(c) “Simplified output” means output composed of fewer dimensions than the respective inputs used to generate it.
(d) “Value-effective” includes, but is not limited to, the following:
(1) The quality and effectiveness of steps taken by the prospective contractor to prevent disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code).
(2) The extent and quality of the internal policy adopted by the prospective contractor for how bias in the automated decision system, is identified and mitigated to prevent disparate impacts on
the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code), and how it will respond to claims or evidence of bias that may arise within the terms of the contract.
12115.3.
On or before January 1, 2023, the department shall establish and make public guidelines for identifying automated decision systems that are subject to the requirements set forth in Section 12115.5. 12115.5 in a manner generally consistent, if appropriate, with international high-risk frameworks and standards.12115.5.
(a) Contract awards for goods or services that include the use, licensing, or development of an automated decision system for a high-risk application shall be based on the proposal that provides the most value-effective solution to the state’s requirements, as determined by the evaluation criteria contained in the solicitation document, and shall be determined based on comprehensive assessment of objective criteria not limited to cost alone.(b) Awarding of contracts subject to this chapter, including, but not limited to, solicitation for acquisitions, evaluation of proposals, and selection of contractors, shall be conducted pursuant to the requirements specified
in Section 12102.2 for awarding contracts based on the proposal that provides the most value-effective solution to the state’s requirements.
(c) A bid response submitted by a prospective contractor for a good or service that includes the use, licensing, or development of an automated decision system for a high-risk application shall not be considered responsive to the solicitation document unless the bid response includes an automated decision system impact assessment that makes the following disclosures to the contracting agency:
(1) Specify the name, vendor, and version of the automated decision system and describe its general capabilities, including, but not limited to, reasonably foreseeable capabilities outside the scope of its proposed use.
(2) Describe the purpose of the automated decision system, including, but not limited to, the decision or decisions it can make or support, and its intended benefits compared to alternatives, including, but not limited to, the results of any research assessing its efficacy and relative benefits.
(3) Provide a thorough explanation of how the automated decision system functions, the logical relationship between data inputs and outputs, and how those outputs relate to the decision or decisions made or supported by the system, including, but not limited to, limitations on inferences that can be drawn from those results.
(4) Describe the affirmative steps the prospective contractor has taken, or any third-party
engagement, to conduct legitimate, independent, and reasonable tests of the automated decision system to help assess any risks posed to the privacy or security of personal information and any risks that may result in inaccurate, unfair, biased, or discriminatory decisions impacting natural persons.
(5) Describe any potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the proposed use of the automated decision system, including, but not limited to, reasonably foreseeable capabilities outside the scope of its proposed use.
(6) Describe any internal policies the prospective contractor has adopted for identifying potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section
51 of the Civil Code) resulting from the proposed use of the automated decision system.
(7) Provide best practices for the proposed high-risk application of the automated decision system to avoid or minimize any disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code), including all of the following:
(A) How and when the automated decision system should be deployed or used, and the relevant technical expertise necessary to minimize the potential for inaccurate, unfair, biased, or discriminatory decisions impacting natural persons.
(B) How to limit the collection and retention of information to that which is directly relevant and necessary for the specified purpose.
(C) How automated decision system data should be stored and accessed to mitigate security risks and threats.
(8) Any additional information specified in the solicitation, or otherwise required by the contracting agency for the purpose of effectively evaluating and avoiding or minimizing disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the use of the automated decision system.
(9) Any additional information required in accordance with regulations adopted by the Department of Technology pursuant to Section 12117.
12116.
A state agency that awards a contract for goods or services that include the use, licensing, or development of an automated decision system for a high-risk application shall, within 10 30 days of awarding that contract, submit to the Department of Technology a high-risk automated decision system accountability report, including, but not limited to, report that includes a clear and understandable statement
of the following:(a) The name, vendor, and version of the automated decision system.
(b) The type or types of data that will be used as inputs for the automated decision system, how that data will be generated, collected, and processed, and the type or types of data the system is likely to generate in the course of its proposed use.
(c) A description of the purpose of the automated decision system, including what decision or decisions it will be used to make or support, and a detailed determination of whether, and how, the system serves reasonable objectives and furthers a legitimate interest.
(d) A clear use and data management policy that includes protocols
for the following:
(1) How and when the automated decision system will be deployed or used and by whom, including, but not limited to, the relevant technical expertise of the user or users, the factors that will be used to determine where, when, and how the technology will be deployed, whether the technology will be operated continuously or used only under specific circumstances, if the system will be operated or used by another entity on behalf of the agency, and, if so, an explicit description of the conditions of that entity’s access and applicable protocols.
(2) Any additional rules and processes that will govern the use of the automated decision system.
(3) How automated decision system data will be securely stored and
accessed, whether the agency intends to share access to the automated decision system or the data from that automated decision system with any other entity, and, if so, the purpose for sharing that access or data, the specific entities with whom that access or data will be shared, and the manner in which that access or data will be shared, including, but not limited to, specific protocols to ensure compliance with any applicable privacy and security laws.
(4) How the agency will ensure that all personnel who operate the automated decision system or access its data are knowledgeable about, and able to ensure compliance with, the use and data management policy prior to the use of the automated decision system and throughout its contracted use.
(e) A description of how the agency
will ensure that all personnel responsible for the adoption and operation of the automated decision system understand the system’s decisional criteria, the respective weights of those criteria, and the factors that may affect or underlie specific results the system produces.
(f) A description of any public or community engagement that has been carried out, and any intended future public or community engagement, pertaining to the use of the automated decision system.
(g) A description of any potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the proposed use of the automated decision system, and a detailed mitigation plan for identifying and minimizing the potential for any disparate impacts
throughout the contracted use of the system, including, but not limited to, any procedures to regularly audit its performance.
(h) A description of the fiscal impact of the use, licensing, and deployment of the automated decision system, including, but not limited to, initial acquisition costs, ongoing operating costs such as maintenance, personnel, legal compliance, auditing, data retention, and security costs, and any cost savings that would be achieved through the use of the automated decision system, as well as a comparison with the costs of alternative solutions for achieving the agency’s purposes.
(i) The extent to which members of the public have access to the results of the automated decision system
system, including an explanation for the basis of a resulting decision in terms understandable to a layperson,
and are able to correct or object to its results, and where and how that information will be made available and any applicable procedures for initiating corrections or objections, as appropriate.
(j) Any other information the Department of Technology determines to be reasonably necessary to carry out the provisions of this chapter.
12116.5.
(a) Within 30 days of a contract award by a state agency for goods or services that include the use, licensing, or development of an automated decision system for a high-risk application, and for the duration of that contract, the Department of Technology shall publish on its internet website the automated decision system impact assessment submitted pursuant to subdivision (c) of Section 12115.5 and the report prepared pursuant to Section 12116.(b) This section shall not be construed to require the publication of trade secrets, as defined in Section 3426.1 of the Civil Code.
12117.
The Department of Technology may adopt regulations and publish guidelines as necessary to effectuate the purposes of this chapter.