Artificial intelligence (AI) and machine learning (ML) have significantly influenced data-driven research, offering both new insights and challenges to scientific replicability and social relevance. This paper critically examines the role of AI as a learning collaborator and the implications of data-driven methods for scientific knowledge production. Using a case study on curb ramp classification in Seattle, WA, it evaluates how different data inputs impact ML classifications and explores the consequences of fitting traditional research design elements to available data and algorithms particularly in the areas of conceptualization, operationalization, and measurement. The study highlights the challenges posed by the black-box nature of AI, the difficulty of achieving replicability in data-driven paradigms, and the necessity of reflexivity in research processes. Results from a series of Random Forest models reveal the tension between accuracy, replicability, and social relevance, emphasizing the importance of transparent, reflexive AI methodologies. The paper argues for a reorientation of AI-driven research towards relevance, advocating for careful consideration of data histories and conceptual rigor to ensure meaningful scientific contributions.