CHAPTER
3.3. Acquisition of Automated Decision Systems for High-Risk Applications
12114.
This chapter shall be known and may be cited as the Automated Decision Systems Accountability Act.12114.5.
It is the intent of the Legislature that agencies of the state use an acquisition method that minimizes the risk of adverse and discriminatory impacts resulting from the design and application of automated decision systems.12115.
For purposes of this chapter, the following shall apply:(a) (1) “Automated decision system” means a computational process, including one derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or
recommendation, that is used to support substantially assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2) “Automated decision system” does not include a tool that does not automate, support, substantially assist or replace human discretionary decisionmaking processes,
processes and that does not materially impact natural persons, including, but not limited to, a junk email filter, firewall, antivirus software, calculator, spreadsheet, database, data set, or other compilation of data.
(b) “High-risk application” means a use of an automated decision system for which any of the following apply: the use of an automated decision system that meets either of the following criteria:
(1)Poses a significant risk to the privacy or security of personal information or is likely to result in inaccurate, unfair, biased, or discriminatory decisions impacting natural persons, taking
into account the novelty of the technology used and the nature, scope, context, and purpose of the automated decision system.
(2)Affects the legal rights, health and well-being, or economic, property, or employment interests of a natural person.
(1) The use of the automated decision system is likely to have a high impact on the legal rights, health, or economic interests of a natural person.
(3)Involves
(2) The use of the automated decision system is likely to pose a material risk of harm from the use of the personal information of a significant number of individuals with regard to race, color, national origin, political opinions, religion, trade union membership, genetic data, biometric data, health, gender, gender identity, sexuality, sexual orientation, criminal record, or any other characteristic identified in the Unruh Civil Rights Act (Section 51 of the Civil Code).
(4)Meets any other criteria established by the Department of
Technology in regulations issued pursuant to Section 12117.
(c) “Simplified output” means output composed of fewer dimensions than the respective inputs used to generate it.
12115.3.
On or before January 1, 2023, the Department of Technology, in consultation with the Department of General Services and with stakeholder input, shall establish and make public guidelines for identifying automated decision systems that are subject to the requirements set forth in Section 12115.5 in a manner generally consistent, if appropriate, with international high-risk frameworks and standards.12115.4.
(a) On or before June 30, 2023, the Department of Technology shall conduct a comprehensive inventory of all high-risk automated decision systems that have been proposed for, or are being used, developed, or procured by, state agencies. The department shall submit a report of the comprehensive inventory to the Legislature by July 31, 2023.(b) The Department of Technology shall repeat the process specified in subdivision (a) on or before June 30, 2025, and on or before June 30, 2027.
(b)
(c) The report required by this section shall be submitted in compliance with Section 9795 of the Government Code.
(c)
(d) Pursuant to Section 10231.5 of the Government Code, this section is repealed on July 31, 2027.
12115.5.
Beginning January 1, 2023, the Department of Technology or any other state agency seeking to award a contract for goods or services that includes the use, licensing, or development of an automated decision system for a high-risk application shall encourage a bid response submitted by a prospective contractor to include an automated decision system impact assessment report that makes the following disclosures to the contracting agency:(a) Specify the name, vendor, and version of the automated decision system and describe its general capabilities,
capabilities and limitations, including, but not limited to, reasonably foreseeable capabilities outside the scope of its proposed use.
(b) Describe the purpose of the automated decision system, including, but not limited to, the decision or decisions it can make or support, support and its intended benefits compared to alternatives, including, but not limited to, the
results of any research assessing information about its efficacy and relative benefits.
(c) Provide a thorough explanation of how the automated decision system functions, the logical relationship between data inputs and outputs, and how those outputs relate to the decision or decisions made or supported by the system, including, but not limited to, limitations on inferences that can be drawn from those results. outputs.
(d) Describe the affirmative steps the prospective contractor has taken, or
any third-party engagement, to conduct legitimate, independent, legitimate and reasonable tests of the automated decision system to help assess any risks posed to the privacy or security of personal information and any risks that may result in inaccurate, unfair, biased, or discriminatory decisions impacting natural persons.
(e) Describe any potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the proposed use of the automated decision system, including, but not limited to, reasonably foreseeable capabilities outside the scope of its proposed use.
(f) Describe any internal policies the prospective contractor has adopted for identifying potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section
51 of the Civil Code) resulting from the proposed use of the automated decision system.
(g) Provide best practices for the proposed high-risk application of the automated decision system to avoid or minimize any disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code), including all of the following:
(1)How how
and when the automated decision system should be deployed or used, used and the relevant technical expertise necessary to minimize the potential for inaccurate, unfair, biased, or discriminatory decisions impacting natural persons.
(2)How to limit the collection and retention of information to that which is directly relevant and necessary for the specified purpose.
(3)How automated decision system data should be stored and accessed to mitigate security risks and threats.
(h) Any additional information specified in the solicitation, or otherwise required by the contracting agency for the purpose of effectively evaluating and avoiding or minimizing disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the use of the automated decision system.
(i) Any additional information required in accordance with regulations adopted by the Department of Technology pursuant to Section 12117.
12116.
On and after January 1, 2023, a state agency that awards a contract for goods or services that includes the use, licensing, or development of an automated decision system for a high-risk application shall, within 30 days of awarding that contract, submit to the Department of Technology a copy of the automated decision system impact assessment report, if any, included in the bid response pursuant to Section 12115.5 that also includes a clear and understandable statement of the following: (a) The extent to which members of the public have access to the results of the automated decision system, including an explanation for the basis of a resulting decision in terms understandable
to a layperson, and are able to correct or object to its results, and where and how that information will be made available and any applicable procedures for initiating corrections or objections, as appropriate.
(b) Any other information the Department of Technology determines to be reasonably necessary to carry out the provisions of this chapter.
12116.5.
(a) Within 30 days of a state agency’s submission of an automated decision system impact assessment report, the Department of
Technology may publish the report on its internet website.(b) This section shall not be construed to require the publication of trade secrets, as defined in Section 3426.1 of the Civil Code. If a prospective contractor or third-party vendor discloses any proprietary information or intellectual property to the Department of Technology, the proprietary information or intellectual property shall be kept strictly confidential and shall not be subject to public disclosure.
12117.
On or before January 1, 2023, the Department of Technology shall develop a sample automated decision system impact assessment report for prospective contractors and may shall adopt regulations and publish guidelines as necessary to effectuate the purposes of this chapter and shall do so in a manner consistent, where possible, with international high-risk frameworks and impact assessment requirements.