Amended  IN  Assembly  March 25, 2021

CALIFORNIA LEGISLATURE— 2021–2022 REGULAR SESSION

Assembly Bill
No. 13

Introduced by Assembly Member Chau

December 07, 2020

An act to add Title 1.81.8 (commencing with Section 1798.400) to Part 4 of Division 3 of the Civil Code, Section 2004 to, and to add Chapter 3.3 (commencing with Section 12114) to Part 2 of Division 2 of, the Public Contract Code, relating to automatic decision systems.


LEGISLATIVE COUNSEL'S DIGEST


AB 13, as amended, Chau. Personal rights: Public contracts: automated decision systems.
Existing law governing the acquisition of information technology goods and services requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by, or under the supervision of, the Department of Technology, as prescribed. Existing law requires all other contracts for the acquisition of information technology goods or services, whether by lease or purchase, to be made by or under the supervision of the Department of General Services. Existing law grants the Department of Technology or the Department of General Services final authority in the determination of information technology procurement procedures applicable to specified acquisitions. Existing law grants the Department of Technology the final authority in the determination of information technology procurement policy.
Existing law states the intent of the Legislature that those policies and procedures developed by the Department of Technology and the Department of General Services provide for, among other things, the expeditious and value-effective acquisition of information technology goods and services to satisfy state requirements. Existing law provides that, for these purposes, “value-effective acquisition” may be defined to include, among other things, the operational cost that the state would incur if the bid or proposal is accepted, the quality of the product or service, or its technical competency, and innovative use of current technologies and quality results.
Existing law requires contract awards for all large-scale systems integration projects to be based on the proposal that provides the most value-effective solution to the state’s requirements, as determined by the evaluation criteria contained in the solicitation document. Existing law requires evaluation criteria for the acquisition of information technology goods and services, including systems integration, to provide for the selection of a contractor on an objective basis not limited to cost alone. Existing law requires specified processes and procedures for the solicitation for acquisitions, the evaluation of proposals, the selection of contractors, and the consideration of protests by participating bidders.
This bill would enact the Automated Decision Systems Accountability Act of 2021 and state the intent of the Legislature that state agencies use an acquisition method that minimizes the risk of adverse and discriminatory impacts resulting from the design and application of automated decision systems. The bill would define “automated decision system” for purposes of the bill’s provisions to mean a computational process, including one derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues a score, classification, recommendation, or other simplified output that is used to support or replace human decisionmaking and materially impacts natural persons.
The bill would require contract awards for goods or services that include the use, licensing, or development of an automated decision system for a high-risk application, as defined, to be based on the proposal that provides the most value-effective solution to the state’s requirements, as determined by the evaluation criteria contained in the solicitation document, and to be determined based on comprehensive assessment of objective criteria not limited to cost alone. The bill would require awarding of contracts subject to the bill’s provisions to be conducted pursuant to the requirements referenced above for awarding contracts for large-scale integration projects based on the proposal that provides the most value-effective solution to the state’s requirements. Under the bill, to be considered responsive to a solicitation for these contract, a bid response would be required to include an automated decision system impact assessment that makes certain disclosures, including, among others, certain tests of the system to help assess risks posed to the privacy or security of personal information and risks that may result in inaccurate, unfair, biased, or discriminatory decisions impacting natural persons.
The bill would require the Department of Technology, on or before January 1, 2023, to establish and make public guidelines for identifying automated decision systems that are subject to the bill’s requirements. The bill would require a state agency to submit to the department, within 10 days of awarding such a contract, a high-risk automated decision system accountability report that includes, among other things, a description of any potential disparate impacts, as specified, from the proposed use of the automated decision system, and a detailed mitigation plan for identifying and minimizing the potential for any disparate impacts throughout the contracted use of the system. The bill would require, within 30 days of such a contract award and for the duration of the contract, the department to publish on its internet website the automated decision system impact assessment submitted by the contractor and the high-risk automated decision system accountability report prepared by the state agency. The bill would authorize the department to adopt regulations and publish guidelines as necessary to effectuate the purposes of the bill.
The bill would authorize a local agency, for a contract for a good or service that includes the use, licensing, or development of an automated decision system for a high-risk application, to require a bid response submitted by a prospective contractor to include an automated decisions system impact assessment in order to be considered responsive to the solicitation. The bill would also authorize the local agency to base the contract award on the proposal that provides the most value-effective solution to the agency’s requirements pursuant to the above-described provisions of the bill.

Existing law establishes the Department of Financial Protection and Innovation, headed by the Commissioner of Financial Protection and Innovation. Under existing law, the department has charge of the execution of specified laws relating to various financial institutions and financial services.

Existing law, the California Fair Employment and Housing Act, protects and safeguards the right and opportunity of all persons to seek, obtain, and hold employment without discrimination, abridgment, or harassment on account of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, or military and veteran status.

Existing law regulates the use of personal information, including the California Consumer Privacy Act of 2018, which grants a consumer various rights with regard to personal information relating to that consumer that is held by a business. The act requires a business that collects personal information about a consumer to disclose the consumer’s right to delete personal information in a form that is reasonably accessible to consumers and in accordance with a specified process.

This bill would enact the Automated Decision Systems Accountability Act of 2021. The bill would require a business in California that provides a person, as defined, with a program or device that uses an automated decision system (ADS) to take affirmative steps to ensure that there are processes in place to continually test for biases during the development and usage of the ADS, conduct an ADS impact assessment on its program or device to determine whether the ADS has a disproportionate adverse impact on a protected class, as specified, examine if the ADS in question serves reasonable objectives and furthers a legitimate interest, and compare the ADS to alternatives or reasonable modifications that may be taken to limit adverse consequences on protected classes. The bill would require a business, by March 1, 2023, and annually thereafter, to submit a report to the Department of Financial Protection and Innovation providing specified information about its ADS impact assessment. The bill would also require a business, if it makes any significant modification to an ADS, to reconduct an ADS impact assessment under these circumstances.

The bill would require the department, by January 1, 2023, to develop a procedure for businesses to use in making the required reports and to make general information on the reporting process available on its internet website. The bill would require the department, if a business fails to comply with these procedures, to send a written notice to the business regarding its failure to comply, and would require the business to submit the report within 60 days of the date of that notice. The bill would make violations of these provisions subject to a civil penalty.

The bill would also require the department, by March 1, 2023, to establish an Automated Decision Systems Advisory Task Force, composed of various representatives from the public and private sectors, for the purpose of reviewing and providing advice on the use of automated decision systems in businesses, government, and various other settings.

Vote: MAJORITY   Appropriation: NO   Fiscal Committee: YES   Local Program: NO  

The people of the State of California do enact as follows:


SECTION 1.

 Section 2004 is added to the Public Contract Code, to read:

2004.
 (a) For purposes of this section, “value-effective” has the same meaning as that term is defined in Chapter 3.3 (commencing with Section 12114) of Part 2.
(b) Notwithstanding any other provision of law requiring a local agency to award contracts to the lowest responsible bidder, a local agency may, for a contract for a good or service that includes the use, licensing, or development of an automated decision system for a high-risk application, do both of the following:
(1) Require a bid response submitted by a prospective contractor to include an automated decision system impact assessment in order to be considered responsive to the solicitation.
(2) Base the contract award on the proposal that provides the most value-effective solution to the agency’s requirements, pursuant to Chapter 3.3 (commencing with Section 12114) of Part 2.

SEC. 2.

 Chapter 3.3 (commencing with Section 12114) is added to Part 2 of Division 2 of the Public Contract Code, to read:
CHAPTER  3.3. Acquisition of Automated Decision Systems for High-Risk Applications

12114.
 This chapter shall be known and may be cited as the Automated Decision Systems Accountability Act of 2021.

12114.5.
 It is the intent of the Legislature that agencies of the state use an acquisition method that minimizes the risk of adverse and discriminatory impacts resulting from the design and application of automated decision systems.

12115.
 For purposes of this chapter, the following shall apply:
(a) “Automated decision system” means a computational process, including one derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues a score, classification, recommendation, or other simplified output that is used to support or replace human decisionmaking and materially impacts natural persons.
(b) “High-risk application” means a use of an automated decision system for which any of the following apply:
(1) Poses a significant risk to the privacy or security of personal information or has the potential to result in inaccurate, unfair, biased, or discriminatory decisions impacting natural persons, taking into account the novelty of the technology used and the nature, scope, context, and purpose of the automated decision system.
(2) Affects the legal rights, health and well-being, or economic, property, or employment interests of a natural person, or otherwise significantly impacts a natural person.
(3) Involves the personal information of a significant number of individuals with regard to race, color, national origin, political opinions, religion, trade union membership, genetic data, biometric data, health, gender, gender identity, sexuality, sexual orientation, criminal record, or any other characteristic identified in the Unruh Civil Rights Act (Section 51 of the Civil Code).
(4) Meets any other criteria established by the Department of Technology in regulations issued pursuant to Section 12117.
(c) “Simplified output” means output composed of fewer dimensions than the respective inputs used to generate it.
(d) “Value-effective” includes, but is not limited to, the following:
(1) The quality and effectiveness of steps taken by the prospective contractor to prevent disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code).
(2) The extent and quality of the internal policy adopted by the prospective contractor for how bias in the automated decision system, is identified and mitigated to prevent disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code), and how it will respond to claims or evidence of bias that may arise within the terms of the contract.

12115.3.
 On or before January 1, 2023, the department shall establish and make public guidelines for identifying automated decision systems that are subject to the requirements set forth in Section 12115.5.

12115.5.
 (a) Contract awards for goods or services that include the use, licensing, or development of an automated decision system for a high-risk application shall be based on the proposal that provides the most value-effective solution to the state’s requirements, as determined by the evaluation criteria contained in the solicitation document, and shall be determined based on comprehensive assessment of objective criteria not limited to cost alone.
(b) Awarding of contracts subject to this chapter, including, but not limited to, solicitation for acquisitions, evaluation of proposals, and selection of contractors, shall be conducted pursuant to the requirements specified in Section 12102.2 for awarding contracts based on the proposal that provides the most value-effective solution to the state’s requirements.
(c) A bid response submitted by a prospective contractor for a good or service that includes the use, licensing, or development of an automated decision system for a high-risk application shall not be considered responsive to the solicitation document unless the bid response includes an automated decision system impact assessment that makes the following disclosures to the contracting agency:
(1) Specify the name, vendor, and version of the automated decision system and describe its general capabilities, including, but not limited to, reasonably foreseeable capabilities outside the scope of its proposed use.
(2) Describe the purpose of the automated decision system, including, but not limited to, the decision or decisions it can make or support, and its intended benefits compared to alternatives, including, but not limited to, the results of any research assessing its efficacy and relative benefits.
(3) Provide a thorough explanation of how the automated decision system functions, the logical relationship between data inputs and outputs, and how those outputs relate to the decision or decisions made or supported by the system, including, but not limited to, limitations on inferences that can be drawn from those results.
(4) Describe the affirmative steps the prospective contractor has taken, or any third-party engagement, to conduct legitimate, independent, and reasonable tests of the automated decision system to help assess any risks posed to the privacy or security of personal information and any risks that may result in inaccurate, unfair, biased, or discriminatory decisions impacting natural persons.
(5) Describe any potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the proposed use of the automated decision system, including, but not limited to, reasonably foreseeable capabilities outside the scope of its proposed use.
(6) Describe any internal policies the prospective contractor has adopted for identifying potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) resulting from the proposed use of the automated decision system.
(7) Provide best practices for the proposed high-risk application of the automated decision system to avoid or minimize any disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code), including all of the following:
(A) How and when the automated decision system should be deployed or used, and the relevant technical expertise necessary to minimize the potential for inaccurate, unfair, biased, or discriminatory decisions impacting natural persons.
(B) How to limit the collection and retention of information to that which is directly relevant and necessary for the specified purpose.
(C) How automated decision system data should be stored and accessed to mitigate security risks and threats.
(8) Any additional information specified in the solicitation, or otherwise required by the contracting agency for the purpose of effectively evaluating and avoiding or minimizing disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the use of the automated decision system.
(9) Any additional information required in accordance with regulations adopted by the Department of Technology pursuant to Section 12117.

12116.
 A state agency that awards a contract for goods or services that include the use, licensing, or development of an automated decision system for a high-risk application shall, within 10 days of awarding that contract, submit to the Department of Technology a high-risk automated decision system accountability report, including, but not limited to, a clear and understandable statement of the following:
(a) The name, vendor, and version of the automated decision system.
(b) The type or types of data that will be used as inputs for the automated decision system, how that data will be generated, collected, and processed, and the type or types of data the system is likely to generate in the course of its proposed use.
(c) A description of the purpose of the automated decision system, including what decision or decisions it will be used to make or support, and a detailed determination of whether, and how, the system serves reasonable objectives and furthers a legitimate interest.
(d) A clear use and data management policy that includes protocols for the following:
(1) How and when the automated decision system will be deployed or used and by whom, including, but not limited to, the relevant technical expertise of the user or users, the factors that will be used to determine where, when, and how the technology will be deployed, whether the technology will be operated continuously or used only under specific circumstances, if the system will be operated or used by another entity on behalf of the agency, and, if so, an explicit description of the conditions of that entity’s access and applicable protocols.
(2) Any additional rules and processes that will govern the use of the automated decision system.
(3) How automated decision system data will be securely stored and accessed, whether the agency intends to share access to the automated decision system or the data from that automated decision system with any other entity, and, if so, the purpose for sharing that access or data, the specific entities with whom that access or data will be shared, and the manner in which that access or data will be shared, including, but not limited to, specific protocols to ensure compliance with any applicable privacy and security laws.
(4) How the agency will ensure that all personnel who operate the automated decision system or access its data are knowledgeable about, and able to ensure compliance with, the use and data management policy prior to the use of the automated decision system and throughout its contracted use.
(e) A description of how the agency will ensure that all personnel responsible for the adoption and operation of the automated decision system understand the system’s decisional criteria, the respective weights of those criteria, and the factors that may affect or underlie specific results the system produces.
(f) A description of any public or community engagement that has been carried out, and any intended future public or community engagement, pertaining to the use of the automated decision system.
(g) A description of any potential disparate impacts on the basis of characteristics identified in the Unruh Civil Rights Act (Section 51 of the Civil Code) from the proposed use of the automated decision system, and a detailed mitigation plan for identifying and minimizing the potential for any disparate impacts throughout the contracted use of the system, including, but not limited to, any procedures to regularly audit its performance.
(h) A description of the fiscal impact of the use, licensing, and deployment of the automated decision system, including, but not limited to, initial acquisition costs, ongoing operating costs such as maintenance, personnel, legal compliance, auditing, data retention, and security costs, and any cost savings that would be achieved through the use of the automated decision system, as well as a comparison with the costs of alternative solutions for achieving the agency’s purposes.
(i) The extent to which members of the public have access to the results of the automated decision system and are able to correct or object to its results, and where and how that information will be made available and any applicable procedures for initiating corrections or objections, as appropriate.

12116.5.
 (a) Within 30 days of a contract award by a state agency for goods or services that include the use, licensing, or development of an automated decision system for a high-risk application, and for the duration of that contract, the Department of Technology shall publish on its internet website the automated decision system impact assessment submitted pursuant to subdivision (c) of Section 12115.5 and the report prepared pursuant to Section 12116.
(b) This section shall not be construed to require the publication of trade secrets, as defined in Section 3426.1 of the Civil Code.

12117.
 The Department of Technology may adopt regulations and publish guidelines as necessary to effectuate the purposes of this chapter.

SECTION 1.

The Legislature finds and declares all of the following:

(a)State law protects the rights of all persons in a variety of contexts without discrimination on account of certain protected characteristics, such as on the basis of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, or marital status, among other characteristics, as described in Section 51 of the Civil Code.

(b)The rise of big data has raised concerns about the use of algorithmic or automated decision systems to make hiring and other workplace decisions, eligibility decisions, insurance eligibility, lending decisions, and marketing decisions quickly, automatically, and fairly.

(c)If the underlying data used for an algorithm or automated decision system is biased, incomplete, or discriminatory, the decisions made by using such devices has the potential to result in massive inequality.

(d)The state has a legitimate and substantial interest in ensuring that automated decision systems used do not result in discrimination.

(e)Therefore, the Legislature finds that it is necessary to require a review of the use of algorithmic decision systems also known as automated decision systems (ADS) in order to detect and prevent discrimination.

SEC. 2.Title 1.81.8 (commencing with Section 1798.400) is added to Part 4 of Division 3 of the Civil Code, to read:
1.81.8.Automated Decision Systems Accountability Act of 2021
1798.400.

This act shall be known and may be cited as the Automated Decision Systems Accountability Act of 2021.

1798.401.

For the purposes of this title, the following definitions apply:

(a)“Automated decision system” or “ADS” means a computational process, including one derived from machine learning, statistics, or other data processing or artificial intelligence techniques, that makes a decision or facilitates human decision making, that impacts persons.

(b)“Automated decision system impact assessment report” or “ADS impact assessment report” means a report containing, but not limited to, the content enumerated in paragraph (3) of subdivision (a) of Section 1798.402.

(c)“Business” means a digital or software company that creates or distributes an ADS.

(d)“Department” means the Department of Financial Protection and Innovation.

(e)“Person” means an individual, firm, association, organization, partnership, limited liability company, business trust, corporation, or public entity of any kind.

1798.402.

(a)A business in California that provides a person with a program or device that uses an ADS shall do all of the following:

(1)Take affirmative steps to ensure that there are processes in place to continually test for biases during the development and usage of the ADS.

(2)Conduct an assessment on its program or device that uses an ADS to do all of the following:

(A)Determine whether the ADS under review has a disproportionate adverse impact on a protected class, as described in subdivision (b) of Section 51. A business may contract with a third party to independently create the ADS impact assessment for the purpose of providing an additional level of credibility.

(B)Examine if the ADS in question serves reasonable objectives and furthers a legitimate interest.

(C)Compare the ADS to alternatives or reasonable modifications that may be taken to limit adverse consequences on protected classes.

(3)On or before March 1, 2023, and annually thereafter, a business shall submit an ADS impact assessment report to the department, in a format developed by the department pursuant to subdivision (b), which includes all of the following:

(A)The name, vendor, and version of the automated decision system and a description of the general capabilities of the automated decision system, including reasonably foreseeable capabilities outside the scope of its designed use.

(B)The type or types of data inputs that the technology uses; how those data are generated, collected, and processed; and the type or types of data the system is reasonably likely to generate.

(C)A description of the purpose of the automated decision system, including what decision or decisions it supports, and its intended benefits, including any data or research demonstrating those benefits, relative to other automated and nonautomated approaches.

(D)A clear use and data management policy, including protocols for all of the following:

(i)How and when the automated decision system can be deployed or used and by whom.

(ii)What practices are in place in order to limit the collection and retention of information to that which is directly relevant and necessary for the specified purpose.

(iii)What information about the automated decision system is and will be available to consumers, and the extent to which consumers have and will have access to the results of the automated decision system and may correct or object to its results.

(iv)What processes will be required prior to the use of the automated decision system.

(v)How automated decision system data will be stored and accessed to mitigate security risks and threats.

(vi)How the business will ensure that all those who operate the automated decision system or access its data are knowledgeable about and able to ensure compliance with the use and data management policy prior to use of the automated decision system.

(E)A description of any third-party engagement, or action by the business, to conduct legitimate, independent, and reasonable tests of the automated decision system to assess the risks posed to the privacy or security of personal information of consumers and the risks that the automated decision system may result in or contribute to inaccurate, unfair, biased, or discriminatory decisions impacting consumers.

(F)A description of a mitigation plan to address any potential disparate impact of the automated decision system on a protected class.

(4)If a business makes any significant modification to an ADS, the business shall reconduct an ADS impact assessment and resubmit the results of that assessment to the department no later than 60 days from the modification.

(b)On or before January 1, 2023, the department shall develop a procedure, including a form, if necessary, for businesses to use in making the reports required pursuant to this section. The department also shall make general information on the reporting process accessible on its internet website on or before January 1, 2023.

(c)If a business fails to comply with this section, the department shall send a written notice to the business of its failure to comply. The business shall have 60 days from the date of the written notice in which to comply, by completing the report and submitting it to the department. Failure by a business to submit the required report shall result in a civil penalty.

1798.403.

On or before March 1, 2023, the department shall establish an Automated Decision Systems Advisory Task Force for the purpose of reviewing and providing advice on the use of automated decision systems in businesses, government, and various other settings. The task force shall consist of all of the following:

(a)Two representatives from advocacy organizations that represent consumers or protected classes of communities, as described in subdivision (b) of Section 51.

(b)Two members from state or local government agencies.

(c)Two representatives from digital or software companies who use or create automated decision systems.

(d)Two representatives from universities or research institutions with expertise in automated decision systems.